dot Redis 8 is here—and it's open source

Learn more

Redis then & now: Adapting with developers through every era

In my 25 years working in software engineering, I’ve seen several computing paradigms come and go. From client-server to distributed systems, SOA, then cloud, and beyond. What fascinates me isn’t just how architectures evolved, but how developer experiences transformed along with them.

This is also true for Redis, which has been a constant companion throughout much of my career and the careers of many other developers. Redis evolved from being just a caching layer to the comprehensive data platform we know today.

This blog post follows both the broader shifts in computing architecture and Redis’ evolution. Since I’ve built systems in most of these times, I’ve seen how Redis has always solved new problems while keeping its main principles of simplicity and performance. Whether you’re a veteran who remembers pre-Redis struggles or a newcomer curious about why Redis works the way it does, this technological trip down memory lane reveals how yesterday’s pain points became today’s elegant solutions.

What you are about to read isn’t just a history of Redis—but a look at how computing has changed. It also looks at how one technology has both adapted to and helped shape that change.

Client-server era (mid-1990s)

In the mid-90s, client-server architecture went mainstream, dividing computational responsibilities between powerful central servers and lighter client machines. Businesses were rapidly digitalizing internal processes while the early commercial internet was taking shape. Developers during this time were building desktop apps using IDEs like Visual Basic, Borland Delphi/C++ Builder, and PowerBuilder. Desktop apps were how the client-server architecture became notably popular, though web apps are also considered a client-server architecture as well.

Building web apps using the client-server architecture proved to be the main source of problems at the time. Given the stateless nature of the HTTP protocol, building something as simple as a shopping cart required complex synchronization between the client-side JavaScript (still in its infancy then) and server-side session stores. Database connections quickly emerged as a significant bottleneck. As user counts grew, apps would exhaust available connections, resulting in system failures. After all, one thing is building a desktop app to handle 25 to 75 users concurrently. The same practices and protocols weren’t designed to handle 5K users or more.

Web apps often suffer from rudimentary and prone-to leaks connection management solutions that gradually lead to performance degradation. At the same time, deployment problems were always there. App servers like Apache HTTP Server and IIS used resource-intensive processes per request that would fail under load, which led to the expensive practice of overprovisioning hardware. Chances are, Redis may have learned from this to decide using its scalable single-threaded event loop design that specifically avoids this problem. Data exchange between systems often relied on custom binary protocols or rudimentary string formats, making cross-language interoperability exceptionally difficult.

These fundamental challenges shaped how developers thought about data access for years to come. When Redis was created years later in 2009, its API design would reflect many lessons learned from this era’s challenges. The Redis Serialization Protocol (RESP) offered a simple text-based design with minimal parsing overhead, accidentally solving many of the protocol efficiency issues developers had struggled with. Redis’ connection pooling approach and multiplexing capabilities also helped clean up the connection overload problems that used to drag down early web apps.

Redis’ language-neutral design allowed any programming language to interact with it using simple socket communication, which resolved interoperability challenges that many developers had faced in earlier years. Though Redis was created a decade after the client-server era’s peak, Redis incorporated principles that addressed many of its persistent challenges.

Distributed systems emergence (late 1990s)

As online services grew in popularity, the limitations of traditional client-server implementations became increasingly apparent. In theory, you could scale client-server setups by just adding more backend servers—but in practice, it wasn’t that simple:

  • Scalability limitations: Traditional client-server systems hit performance limits when too many users connect at once, as a direct result of the shared resources approach for disks and servers.
  • Single point of failure: When a client-server system’s main server crashes, all clients connected lose access to the service. Server endpoint transparency wasn’t client-server’s strongest suit.
  • Geographic/Network latency: Users located far from the server experience slow response times due to physical distance. Pushing servers close to some areas meant having a data center as well, which was practically impossible.

Organizations needed architectures that inherently distributed both data and processing across networks. This evolution toward distributed systems introduced concepts like Remote Procedure Calls (RPC), object state replication, and early clustering that went beyond just server clustering. Technologies like Microsoft COM, Java RMI, and CORBA emerged to help developers to invoke objects distributed over a network to perform data processing tasks, with the goal of breaking apart monolithic server architectures into components that could be deployed and scaled independently. The more objects you had deployed over the network, the greater was the architecture’s ability to scale.

I vividly recall, back in 1999, helping a customer in Brazil to build a surveillance system using CORBA. During that time, I was working as a consultant for IONA Technologies, which had one of the most deployed CORBA brokers in the world called Orbix. We used its multicast UDP protocol between the brokers and the clients to process transactions. The idea behind this protocol was to have clients dispatching requests over the network, and one of the distributed objects deployed would process the request. This approach worked well in our test environment and delivered the scalability and fault-tolerance we needed. But, it failed terribly with the production servers because of network partitions we never anticipated. The result? The regions monitored received the wrong detection alerts from other regions, depending on which broker they hit. I could only dream at that time for something like Redis who treats network partitions as an expected condition rather than an exception.

The reality was that our early distributed systems solutions were brittle and complex. We implemented manual database sharding with app-level logic to route queries to the correct database now deployed across different servers. Our load balancers used simplistic round-robin algorithms that didn’t account for varying request complexities. When replication failed, we often had to take complete outages to resynchronize our systems.

An interesting system design that emerged from the distributed systems era was the maintenance of two parallel codebases for the same app. One optimized for reads and another for writes. Why? Because it was perceived that databases would get more reads than writes, and it was simply wiser to use different databases for each. This would be later known as the Command Query Responsibility Segregation (CQRS) pattern. Developers had to manually direct traffic between these two databases. The complexity was overwhelming, and every deployment felt like getting into dark waters without the proper safety equipment. If only they could use the replication model from Redis which provides a read and a write replica by default, pending only you configure it this way.

These important lessons about distributed state, network partitions, and system coordination became foundational knowledge for developers. When Redis emerged in 2009, it took a notably different approach to data distribution. Where CORBA systems often failed catastrophically under network partitions, Redis’ master-replica design with asynchronous replication was built with principles that give systems the ability to survive and recover from such scenarios.

As distributed systems matured in the late 1990s, organizations faced a new challenge: these powerful but complex architectures were creating siloed apps that couldn’t easily share functionality. Business leaders demanded more agility and reusability from their expensive technological investments. This need to break down app silos while maintaining the scalability of distributed systems gave birth to SOA in the early 2000s.

Service-Oriented Architecture (early 2000s)

Service-Oriented Architecture (SOA) introduced loosely-coupled business capabilities exposed as discoverable services. Organizations began decomposing monolithic apps, requiring middleware, enterprise service buses, as well as standardized communication protocols like SOAP. The need for decomposing monolithic apps started with businesses realizing how slow they were to maintain and evolve their systems. By breaking down parts of the business logic into reusable components, they would update them individually and most importantly, reuse them as needed. SOA promised flexibility but delivered complexity instead.

Take WSDL for example. The XML files that described how clients could interact with SOAP web services, quickly grew to thousands of lines for a medium-sized project, becoming essentially unreadable by humans. Developers spent more time debugging XML parsing issues than implementing business logic. Simple service calls that should have taken milliseconds often took seconds due to XML serialization/deserialization overhead, creating a constant performance battle. While JSON is ubiquitous and human-readable today, XML was verbose and required special tools just to inspect.

Service versioning became a particular nightmare. When service contracts changed, teams faced an impossible choice: Break existing clients or maintain multiple service versions indefinitely. Many organizations ended up supporting three or four versions of the same service, dramatically increasing operational complexity. Meanwhile, service discovery mechanisms like UDDI registries proved too cumbersome for practical use. UDDI registries were supposed to provide a central location where users could find the needed services, like a Yellow Pages catalog. Instead, what we found was hard-coded endpoints implemented on a per-project basis pointing to services no one was aware of.

Distributed transactions were the most challenging aspect of SOA implementations. Coordinating operations across multiple services required complex compensation mechanisms or two-phase commits that few teams implemented correctly. The result was data inconsistency that required constant reconciliation processes and manual intervention. Imagine how happy developers during the SOA days would be by using something like MULTI/EXEC commands from Redis.

The middleware meant to facilitate communication between multiple services became a bottleneck of its own. ESBs were the middleware at the time that would serve the purpose of exposing different services over managed endpoints, so that when a request came it would forward to the right backend service, sometimes processing message transformations. Today’s lightweight API gateways serve a similar purpose—but ESBs are far more complex and centralized. Products like Sonic ESB, TIBCO BusinessWorks, BEA WebLogic Integration, and Microsoft BizTalk required entire teams for configuration and maintenance. When these ESBs failed, entire business operations would simply halt, creating the ultimate irony, SOA was supposed to reduce tight coupling. Instead, its middleware created unprecedented levels of it.

Redis emerged in 2009, well after SOA’s peak influence, yet its design principles would represent a philosophical contrast to SOA’s complexity. Where SOA offered heavyweight XML-based SOAP calls taking hundreds of milliseconds, Redis provided simple text commands completed in sub-millisecond times. While SOA required complex ESBs with dedicated administration teams, Redis needed no middleware at all—just a lightweight server process anyone could deploy in minutes. This wasn’t a deliberate rejection of SOA principles, as Redis was created to solve specific data access problems at Salvatore Sanfilippo’s startup rather than as an architectural statement. Nevertheless, Redis’ “composition over complexity” philosophy stood in stark contrast to the monolithic ESBs of the SOA era.

While SOA promised business agility through service reuse, it created a new bottleneck: infrastructure provisioning. Deploying new services or scaling existing ones required weeks of hardware procurement and configuration. This disconnect between SOA’s promise of rapid service composition and the reality of slow infrastructure deployment set the stage for virtualization and cloud computing. These technologies would eventually provide the foundation that made Redis adoption so seamless in cloud environments.

Virtualization & Early cloud (2006-2010)

Hardware virtualization allowed multiple guest operating systems to share physical infrastructure, laying the groundwork for cloud computing. AWS launched S3 and EC2, making on-demand computing resources available. Before AWS, getting a server meant literal physical hardware with weeks of procurement time, not the minutes we are all accustomed to today. Apps began to be designed for fault tolerance and variable infrastructure. These concepts evolved specifically because of cloud computing’s ephemeral nature. After all, we learned a few things from the distributed systems era.

During the early days of cloud architectures, the common struggle was building stateful apps. Database scaling remained a particular pain point. People often reverted to vertical scaling with larger instances rather than implementing true horizontal scaling, which was the whole point of the cloud. It was against this backdrop of emerging cloud computing needs that Redis was born in 2009.

Redis 1.0 was created by Salvatore Sanfilippo primarily to solve practical problems at his startup—not as a strategic response to cloud computing trends, but as a focused solution to specific data handling challenges. Sanfilippo encountered significant problems in scaling certain types of workloads using traditional database systems, which led him to prototype the first proof-of-concept version of Redis. The in-memory design with disk persistence was created to enable extremely fast data access without the disk I/O bottlenecks that limited other data stores. This approach happened to align perfectly with cloud computing’s need for efficient resource utilization. But above all, Redis was created to be simple to use, execute, scale, and operate.

After a few weeks of using the project internally with success, Sanfilippo decided to open-source it, announcing the project on Hacker News. With its initial offering of four core data types:

  • Strings: Beyond simple key-value storage, Redis strings can hold any binary data up to 512 MB with atomic operations. To implement a real-time counter for a dashboard, devs would do it by updating a database record. Under load, this created lock contention and eventually crashed the database. When this was replaced with Redis’ INCR command, the same counter could handle thousands of increments per second with no lock issues, and the dashboard finally showed accurate real-time data.
  • Lists: Implemented as linked lists for efficient head/tail operations with O(1) complexity, this type enabled queue implementations that previously required dedicated message brokers. In the pre-Redis architecture, implementing a simple job queue required setting up a full message broker with exchanges, queues, and bindings, as well as a full day’s work before writing any app code. With Redis Lists, you could implement the same pattern with LPUSH and BRPOP commands in about 10 lines of code. We went from complex infrastructure to a simple, reliable solution that took minutes to implement.
  • Sets: Unordered collections with O(1) membership operations revolutionized how devs handled unique collections. Sets were particularly useful for building use cases like a social feature that needed to track mutual connections between users. Using SQL databases, required complex joins that slowed to a crawl with just a few thousand users. Switching to Redis Sets, we could use SINTER to find mutual connections instantly, and the feature went from being the site’s slowest to its fastest.
  • Sorted sets: Perhaps Redis’ most innovative early data structure, sorted sets assigned scores to members, keeping priority queues, leaderboards, and range queries. Anyone in need of building real-time leaderboards across millions of users was in for a treat. With SQL databases, it would take 30 seconds to calculate top players. After switching to Redis Sorted Sets, the same leaderboard is updated in real time with ZADD and ZRANGE commands. Users went from super frustrated to engaged because they could see their rank change instantly after earning points.

The in-memory design of Redis delivered exceptional performance while its built-in data structures provided versatility beyond simple key-value stores. Developers embraced Redis  for use cases like session management, caching, and real-time analytics, appreciating how it simplified previously complex patterns.

Redis’ architecture directly addressed the client-server era’s need for efficient protocols and the distributed systems requirement for clear consistency guarantees. Its simplicity was the antithesis of the heavyweight middleware developers struggled with during the SOA era, offering a refreshing alternative that aligned with cloud computing’s emphasis on scale, flexibility, and resilience.

Cloud computing solved the infrastructure agility problem that had hampered SOA adoption, but it exposed new challenges in app architecture. Organizations could now provision resources in minutes instead of weeks, but their monolithic apps couldn’t take full advantage of cloud’s elastic scaling capabilities. This limitation, combined with the lessons learned from SOA’s overcomplicated middleware; led to the microservices revolution, an evolution that Redis was uniquely positioned to support with its versatile data structures.

Microservices revolution (2010-2014)

Perhaps the most iconic revolution in the software engineering community started with microservices. The idea was to break monoliths into independently deployable services with bounded contexts. The fundamental difference between microservices and SOA relied on the need for different teams to own their services, as opposed to SOA where central teams were formed to own the services. This is the reason why we say microservices use a decentralized model, whereas SOA uses a centralized one.

This architectural shift asked for new patterns for service discovery, communication, and distributed data management. Sharing databases by itself became an anti-pattern, yet services still needed coordination mechanisms and fast data access. At this point, developers would still be hurt by the mistakes of the SOA era, often thinking microservices were just a rebranded version of it. In a nutshell, there were similarities between both architectural styles. But in reality, they were meant to solve different problems related to code ownership, versioning, and control. Therefore, there was some resistance in the beginning.

The first microservices migration was certainly eye-opening. We went from maintaining one huge monolith with performance issues to orchestrating several microservices that couldn’t reliably communicate. Service A would call Service B, which would call Service C. By the time the response came back to the user, they would already have left the site. We desperately needed something to glue these services together.

Along with the microservices revolution, came the implementation of complex service meshes, API gateways, and custom circuit breakers. REST APIs became the standard way to share data, with JSON as the representation of data entities. While this was great for interoperability, it created a considerable bottleneck with JSON serialization cycles back and forth with databases. Every microservice had its database, which created data integrity and consistency challenges across service boundaries. The distributed systems complexity that was previously hidden in our monolith now became our daily concern.

In 2010, Redis 2.0 added support for Pub/Sub capabilities. This feature became the communication backbone for microservices to exchange data using the messaging pattern. Before Pub/Sub, sharing data between microservices relied on complex messaging technologies. To make them durable, they would often need to enable their persistence capabilities, which ironically would only serve the purpose of slowing the system down. Switching to Redis Pub/Sub allowed the code to become trivial to implement and lightning-fast. Just publish events and subscribe where needed. Microservice implementations went from error-prone and fragile to reliable and easy to extend with new notification types.

To address the emerging needs of microservices architectures, Redis continued evolving its feature set. Still in 2010, Redis introduced support for enhanced replication and the Hashes data type. Hashes, as a map-like structure that dramatically reduces memory usage for object representation, happened to be perfect for the domain model approach that microservices embraced. The replication capabilities strengthened Redis’ fault-tolerance and scalability—two key properties for developers building distributed microservice apps.

In 2012, Redis 2.6 added Lua scripting support, bringing computation closer to data. With this feature, developers could build solutions such as a rate-limiting service that needed to check counters, increment them if below a certain threshold, and return a decision. All of this atomically. Before Lua scripting, this required multiple round-trips to Redis, creating race conditions in high-traffic scenarios. With Lua scripts, this entire logic was encapsulated in a single server-side operation, eliminating the race conditions and reducing latency from 50ms to under 5ms.

The microservices paradigm amplified the SOA-era need for simpler, more performant middleware. Exactly the problem Redis was born to solve. The Redis’ Pub/Sub system provided the decoupling that SOA had promised but with significantly less complexity, while data structures like Hashes offered a proper solution to the state management challenges that developers had struggled with during the early distributed systems era.

As microservices gained traction, a new challenge emerged: consistent deployment across different environments. Development teams found their services worked locally but failed in production due to environmental differences. Additionally, the operational burden of managing hundreds of microservices demanded new approaches. Redis’s growing feature set would soon prove valuable in addressing these deployment consistency and service coordination challenges that containerization aimed to solve.

Containerization wave (2013-2016)

Docker revolutionized app packaging by making containers mainstream. Anyone building microservices at this point would use containers to deploy their code because it became simpler to update and maintain. Apps and dependencies could now be bundled together for better consistency across environments and  higher deployment density. The “build once, run anywhere” container philosophy transformed development workflows. But, managing the state in ephemeral containers presented new challenges.

When I first adopted Docker, stateless apps containerized beautifully, but stateful services like databases and caching systems were much more difficult. I fought with volume mounts, data persistence, and container networking. The ephemeral nature of containers meant instance identities weren’t stable, breaking up many assumptions my apps made about infrastructure.

In 2015, Redis 3.0 added support for Redis Clusters to offer horizontal scaling, crucial for containerized environments where vertical scaling wasn’t always an option. And let’s face it—even if it was—it wouldn’t feel right. Before Redis Cluster, we manually had to shard data across multiple Redis instances using client-side hashing. This required custom code in every app and made resharding nearly impossible without any downtime. With Redis Cluster, we could distribute workloads across multiple containers while maintaining a unified view of the data. When scale was needed, we could simply add nodes dynamically without any app changes. The clustering capabilities introduced in Redis directly addressed the scaling challenges we had faced in early distributed systems, but with an elegance and simplicity that reflected lessons learned from those earlier, more complex approaches.

Although not particularly tied to the containerization era, at this point we noticed a surge of location-based services as mobile apps proliferated. For example, let’s say a developer needed to build a store locator feature to find retail locations within a certain radius of the user. A possible solution was using a PostgreSQL database with PostGIS extensions, requiring complex setup and maintenance. In 2016, Redis 3.2 introduced the geospatial commands. Developers could use the commands GEOADD and GEORADIUS. The implementation was simpler, performance improved by an order of magnitude, and it eliminated an entire database from the infrastructure only for that.

Redis became quite popular with Docker. Redis’ small footprint and fast startup time made it particularly container-friendly, with official Docker images quickly becoming some of the most popular in the ecosystem. Its ability to perform consistently even with constrained resources made it an ideal choice for the high-density, scalable-first, and variable-performance world of containers developers were working in.

Serverless computing (2017-2022)

While containers solved the packaging and deployment challenges of microservices, they introduced new operational complexity. Managing container orchestration, networking, and scaling still required significant expertise. As organizations sought to focus more on business logic and less on infrastructure management, serverless computing emerged as the next logical step. Redis would continue its evolution alongside this architectural shift, becoming a critical piece of the stateful puzzle in a stateless serverless world.

Functions-as-a-Service (FaaS) platforms like AWS Lambda, Azure Functions, and Google Cloud Functions eliminated the need to provision containers, as well as manage scaling thresholds. Event-driven architectures gained popularity alongside serverless adoption. Development focused on small, stateless functions with fast cold start times, creating a paradox: computing became more ephemeral but the need for statefulness was still a thing. It wasn’t hard to find developers struggling with state management challenges.

Functions themselves were stateless by design—but most apps needed state somewhere. Cold starts meant connection pooling didn’t work well. The fifteen-minute maximum execution time limited what task you could execute until its completion. Most importantly, coordinating across function invocations became a significant architectural challenge with many anti-patterns growing in popularity to overcome them. There was a need in this space for scalable event-driven architectures.

In 2018, Redis 5.0 added support for the Streams data type. Streams are a log-style data structure perfectly timed for event-driven architectures. Say you wanted to implement an event-sourcing pattern in a serverless architecture, initially using a combination of SQS queues and Amazon DynamoDB tables. This solution was complex, high latency, and costly. When Redis Streams arrived, developers replaced both services with just Redis. Functions could now publish events and create consumer groups that persisted across invocations. The code was simpler, more reliable, and significantly faster.

Functions in need of clever ways to operate on data at a scale where precision trade-off was allowed benefit from Redis probabilistic data structures. Data types like Bloom Filter, Cuckoo Filter, and Count-Min Sketch helped manage the high-throughput, ephemeral nature of serverless workloads with ease and elegance. For instance, if you needed to prevent duplicate processing in a serverless workflow that handled millions of events daily, the naive solution would be storing all processed IDs in a database. But this would scale poorly and quickly become expensive. With Redis Bloom filters, you could check for duplicates with near-perfect accuracy using less than a fraction of the time. This allowed serverless functions to stay within the timeout limits while handling 10x the previous throughput.

Additional enhancements to the Pub/Sub support of Redis improved event distribution in serverless environments. In 2020, Redis 6.0 added support for client-side tracking, and in 2022, Redis 7.0 added support for sharded pub/sub. These features improved the ability of Redis in being used for serverless environments, offering better performance for high-volume event distribution and pattern-based subscriptions. For instance, let’s say you needed to build a real-time notification system using serverless functions to alert multiple recipients when events occurred. One way to do it was by using a polling mechanism that generated excessive function invocations and had high latency. With the new Pub/Sub support, developers could implement a push-based model where events immediately trigger the appropriate functions. This approach reduces latency from seconds to milliseconds and cuts their function invocation costs by 80%.

AI-native apps (2022-2024)

The serverless computing model dramatically lowered the barrier for developers to deploy compute-intensive workloads on demand. This democratization of computing resources coincided with breakthroughs in large language models and embedding techniques, making AI capabilities accessible to mainstream developers for the first time. Redis had evolved through each previous era, accumulating capabilities that would now position it as an ideal infrastructure component for this new AI-native computing paradigm.

Apps are increasingly incorporating AI capabilities as core components rather than add-ons. Large language models, embedding vectors, and multi-modal systems drive new architectural patterns focused on handling AI workloads efficiently. Now, developers routinely work with vector embeddings, and similarity search, and often require infrastructure to support AI inferencing alongside traditional app logic. Redis is, once again, the perfect infrastructure for this.

As developers began incorporating AI into their apps, they faced new data challenges. Vector embeddings needed specialized storage and search capabilities. LLM outputs needed to be cached for performance. Retrieval-augmented generation patterns required flexible document storage and fast similarity lookups. Multimodal searches require storing both vectors and documents with equal search capability. None of the traditional databases were optimized for these AI-specific needs.

In 2024, Redis marked a watershed moment in Redis evolution, fully integrating previously separate modules directly into the one core known as the Redis Open Source. Along with all the innovations added throughout the years, the following new features are now available for free to use to aid the development of AI-based systems:

  • Vector search: Native support for similarity search on embeddings as it became crucial for AI apps. Before vector search, developers stored vectors in a specialized vector database separate from their app data in Redis. This meant synchronizing between two systems and added significant latency. With the Redis integrated vector search capabilities, it consolidated everything into Redis. Now your app can store embeddings alongside metadata and perform similarity searches in a single operation, dramatically reducing querying complexity and improving performance.
  • JSON: First-class JSON document support with path queries, eliminating the need to flatten documents into hashes, as well as serialize/deserialize on every operation. Before Redis JSON, you either serialized entire documents as strings or flattened them into hashes with complex key naming conventions. With native JSON support, you can use path expressions like $.conversation[0].role to update specific parts of documents without retrieving or parsing the entire object. This simplified the code and improved performance for partial updates.
  • Agent memory: AI apps require both short and long-term memory. Short memory is needed to handle the state between the processing of different tasks. Long-term memory is needed so history from previous conversations can be used as context for new ones. Redis provides the mechanisms for developers building AI apps to handle both types of memory. This is done with integrations built for frameworks like LangChain, LangGraph, LlamaIndex, and RedisVL.
  • Semantic cache: One of the greatest use cases for AI is building apps using LLMs. However, as you may know, LLMs are expensive, and developers often must avoid processing already-answered questions twice to cut costs and save time. Redis provides semantic cache to address this need. This allows developers to reuse answers, even when the questions are not grammatically the same. Similarly, you can use semantic routing. It allows for alternative routes that can answer user’s questions and the decision is taken based on embeddings.

These new features position Redis as a comprehensive data platform while keeping its legendary performance. Redis is a great fit for AI apps that need to manage both traditional and vector data with minimal latency, turning what would be a complex, multi-service setup into something simpler and easier to manage.. With Redis, developers can design AI systems that can also take advantage of the data foundation built over the years.

Redis 8 (2024-2025)

In 2024, we announced Redis 8, the evolution of 15 years of work, and now a free foundation for developers to build fast apps. Redis 8 represents the culmination of Redis’ evolutionary journey based on every computing paradigm since the client-server era in the 90s. It perfectly addresses the fundamental inefficiencies of client-server with its optimized RESP protocol, the data consistency and coordination challenges of distributed systems with its clustering capabilities, the complexity of SOA with its elegant primitives, the state management challenges of cloud and containers with its data structures, and the event-processing needs of serverless. Yes, Redis was created in 2009, but its primitives and design thinking go way back.

Redis 8 embraces the AI revolution by bringing together all the specialized capabilities developers need in a single, integrated platform. Rather than needing separate systems for different aspects of modern apps, Redis 8 brings these capabilities together while maintaining the performance and simplicity that made Redis beloved in the first place.

Key features that complete Redis’s evolution into a comprehensive data platform include:

  • Multiple data types: Redis 8 provides a complete set of data types that will help developers solve major problems for today’s needs. It includes the known basic types like Strings, Lists, Sets, Sorted Sets, Streams, and Geospatial. But includes specialized ones like Bitmaps, Bitfields, HyperLogLog, Bloom Filter, Cuckoo Filter, T-Digest, Top-K, and Count-Min Sketch. It also includes JSON, Time Series, and Vectors.
  • Hash-field expiration: One of the most popular use cases for Redis is session management. A typical way to manage a session is using the Hashes data type. Before Redis 8, expiring was limited to keys only, which means the entire Hash would need to expire using TTLs. Now, it is possible to expire individual fields of the Hash, making Redis even more attractive for session management, as it grants more control over user session data.
  • Redis Query Engine: The Redis Query Engine, gives you query capabilities for data types like JSON and Vectors using the same API. Developers can create multiple indexes on top of their data so any field becomes searchable. It unlocks use cases in need of vector search, full-text search, geospatial queries, JSON/Hash document queries, and aggregations. All of this while keeping Redis’ performance fast by operations executing in memory.

Redis 8 also represents the most significant performance leap in Redis history, with unprecedented latency improvements across a wide range of commands. Throughout the three milestone releases (M01, M02, and M03), Redis 8 has delivered substantial performance enhancements compared to Redis 7.2.5, with latency reductions ranging from 5.4% to 87.4% across different data structures and commands. The improvements affect over 20 commonly used commands, with a median latency reduction of 16.7% in benchmark testing. Popular commands such as ZADD show up to 36% lower latency, SMEMBERS up to 28%, and HGETALL up to 10%, ensuring that approximately 70% of Redis users will experience significant performance gains.

Redis 8’s new asynchronous I/O threading implementation revolutionizes multi-core performance by assigning clients to specific I/O threads that handle socket reading, writing, and command parsing. When configured with 8 threads on multi-core Intel CPUs, throughput improves by 37% to 112% depending on the commands executed. This advancement makes Redis 8 more CPU efficient and cache-friendly by reducing CPU cycles, cutting memory allocations, and optimizing data access patterns in commonly used code. This means your Redis deployments can now efficiently use all available CPU cores without complex configuration, effectively giving you more power from the same hardware.

A completely redesigned replication mechanism in Redis 8 introduces the concept of parallel replication streams, allowing simultaneous transfer of both the primary node and ongoing changes. This innovation enables primary nodes to handle operations at a 7.5% higher rate during replication, reduces memory buffer requirements on the primary by 35%, and accelerates the replication process by 18%. Testing with a 10GB dataset showed significantly improved robustness when handling simultaneous write operations during replication.

Redis 8 brought performance improvements for its vector implementation as well. With a laser focus on real-world AI apps like semantic search, recommendation engines, and content moderation, Redis was improved to work at a production scale without requiring specialized infrastructure or compromising on response times. This was showcased with a remarkable billion-scale vector search benchmark. Using a dataset of one billion 768-dimensional vectors with FLOAT16 precision (derived from the LAION-5B dataset), Redis demonstrated real-time vector search at an unprecedented scale.

The system achieved 90% precision with a median latency of 200ms and 95% precision with a median latency of 1.3 seconds while executing 50 concurrent search queries. For high-throughput indexing, Redis 8 sustained 66K vector insertions per second with high precision configurations (M=16, EF_CONSTRUCTION=32), and up to 160K writes per second with configurations optimized for speed over precision. All made possible by the newly democratized RQE scaling capabilities.

These performance improvements from Redis 8 aren’t just impressive numbers on a benchmark. They translate to cost savings and better user experiences. Apps respond faster while using fewer resources, which means lower cloud bills and happier users. They represent Redis’ response to the real-world challenges devs have faced throughout computing history. The optimized memory usage addresses resource constraints we struggled with since the client-server days. The enhanced throughput capabilities solve scalability challenges that plague distributed systems.

The asynchronous I/O threading tackles concurrency issues that for a long time hampered microservices. The parallel replication takes on reliability concerns from the cloud era. And the vector search optimizations enable AI capabilities that would have seemed like science fiction just a decade ago. What started as a simple in-memory caching has evolved into a comprehensive data platform solution that addresses the accumulated challenges of three decades of computing evolution.

Redis 8: A platform for every era

Looking back on this journey through computing paradigms, what strikes me most is how Redis has remained a constant companion to developers, adapting to each era’s challenges while staying true to its founding principles. From solving client-server bottlenecks to addressing distributed systems’ state coordination challenges, from simplifying microservices communication to enabling AI apps, Redis has consistently offered elegant solutions to the most pressing needs of each technological age.

Redis 8 is the culmination of this evolutionary journey, integrating vector search for AI, native JSON for complex documents, time series for temporal data, and sophisticated probabilistic structures for massive-scale analytics. In today’s landscape of AI-native apps and complex distributed systems, Redis gives you exactly what developers need: powerful capabilities without complexity.

I’ve watched Redis transform from a simple caching layer to a comprehensive data platform that makes the impossible seem routine. And the best part? This journey is accessible to everyone. Redis Open Source offers all these capabilities for free. Whether you’re building your first app or your five-hundredth, Redis 8 awaits to make your development experience more joyful and productive. Try it today. Your future self will thank you, just as I thank my past self for discovering Redis years ago.