Global real estate company accelerates searches and reduces software licensing costs by $150,000 per month with Elastic Observability

With a presence in more than 110 countries and territories, this leading player in the global real estate franchise landscape boasts over 140,000 agents operating across nearly 9,000 offices.

The parent business is focused on delivering exceptional service to franchisees and their customers. It must also maximize efficiency and drive down operations costs, especially at a time when the real estate industry is under pressure from many directions.

The Data Engineering team aimed to simplify data ingestion from 450 geographically distributed real estate Multiple Listing Services (MLS) across the US and Canada to support these goals. It also wanted to address data flow inefficiencies within the data architecture, consisting of microservices deployed on Kubernetes, with Confluent as the central hub for data exchange.

The director of data engineering at the company says, "The sheer volume of data made identifying the root cause of processing issues like searching for a needle in a haystack. The situation escalated until it severely impacted processing times, leading to backlogs of hours."

The team saw the potential to migrate to Elastic Observability when the incumbent Datadog logging and application performance monitoring (APM) contract came up for renewal.

The most significant factor was cost. The senior data engineer at the company says, "Our team had existing expertise with Elasticsearch since it is the foundation of our website search. Using Elastic Observability meant that we avoided migrating to an entirely new platform along with significant workforce costs and additional licensing fees."

The second factor hinged on the value proposition of Elastic Observability. The team was impressed by rapid feature development, especially with regard to Elastic’s leading-edge machine learning and artificial intelligence technology. "Ultimately, Elastic provided access to the functionalities we truly needed at a lower cost point," says the senior data engineer.

The third differentiating factor was the exceptional support provided by a dedicated Elastic team. "Their approach was devoid of pushy sales tactics," says the senior data engineer. "They consistently demonstrated a willingness to collaborate and provide support for our specific needs."

Today, Elastic Observability monitors all the components of the Kubernetes environment, including most microservices as well as the Confluent data pipeline. The director of data engineering says, "Seamless integration with open source tools allows for data ingestion from virtually any source. We have written custom Logstash plugins that transform data from all 450 sources into the one schema we need to run our website."

This highly flexible Elastic environment means that the company is always up to date with the listings hosted by its franchisees. "We're adding thousands of listings daily that include an average of 25-35 images per listing. Without the efficiency and adaptability of Elastic, we would face the daunting task of building and maintaining a custom ETL solution," says the director of data engineering.

For the director of data engineering, Elastic Observability offers much more than just smoother logging integrations. "The addition of Elastic APM is a paradigm shift for the business," says the senior data engineer. "It's taken us to a whole new level with interactive dashboards, unified observability, and far greater accuracy."

In one example, an engineer wanted to remove rarely used middleware endpoints. "Traditional logging would've been an ordeal sifting through commands, queries, and time frames. With Elastic, it took just three minutes to identify the unnecessary code and reduce our API footprint," says the director of data engineering.

The implementation of observability has also enhanced the ability to interact with logs and application data. A key benefit is the utilization of Spaces within Kibana that allows the company to create customized dashboards and ensure that separate groups can access relevant data without information overload.

The initial adoption of APM also revealed hidden performance issues. "Undetected problems can significantly impact code quality and deployment efficiency," says the director of data engineering. After implementing Elastic APM, the company identified and addressed numerous issues of which it was previously unaware. "This focus on proactive problem identification has resulted in demonstrably improved code quality, faster deployments, and fewer performance-related issues," he says.

As well as operational benefits, Elasticsearch makes it simpler and faster for end users to search real estate listings based on a dataset of 2.5 million unmarked properties alongside a staggering 200 million off-market parcels. The company relies on autocomplete address functionality to help end users easily find the right property.

"We have one essential requirement: lightning-fast address autocomplete within a strict 10-millisecond time frame," says the director of data engineering. "This is a non-negotiable for us, and Elastic delivers without fail."

By adopting Elastic as its observability and APM solution, the company has made significant savings at a time when budget efficiency is a priority for the real estate sector. Had it renewed its license with Datadog, it would be spending about US$150,000 per month.

These efficiencies continue to ripple through the organization. "With Elastic's license, we don't have to pay additional dollars if we want to use a new feature or implement APM into another application," says the senior data engineer. The company also has more control over how long it can maintain log and APM data, going from two weeks to 30 days with Elastic.

Closer monitoring of log data volumes also drives important savings. "Previously, we were about 20% over our contract rate because nobody was watching our login rates," says the senior data engineer. By ingesting the relevant data into Elastic Observability and creating a dedicated dashboard, the company reduced flow through Confluent by 90%. This systematic approach has enabled the business to address bottlenecks and reduce delays across the system.

Elastic also liberates developers from tedious manual log investigations. Elastic APM streamlines this process by enabling faster and more efficient identification of issues. Developers can pinpoint problems quickly and return to core coding tasks sooner.

Another advantage of running Elastic Observability on Elastic Cloud lies in its inherent manageability. Eliminating the need to manage upgrades and routine maintenance tasks enables the company to minimize infrastructure management overhead and frees up valuable technical resources. "My primary concern lies with maximizing team productivity," says the director of data engineering. "Partnering with Elastic allows us to optimize staff allocation and expertise."

Looking to the future, the company is exploring participation in the preview program for Elastic's new serverless offering. The senior data engineer is also excited about the prospect of incorporating Elastic generative AI features into the company's platform. He envisions a future where users can express their search preferences through natural language.

"Instead of filters, imagine simply typing or speaking a prompt like 'I'm looking for a house in Denver with a certain number of bathrooms near this type of establishment,' and the AI search engine intelligently interprets and fulfills the request," says the senior data engineer. "That's the future of the industry."

The integration of language models and a prompt-based search approach supported by Elastic generative AI features has the potential to make this vision reality. "We are enthusiastic about the prospect of leveraging these advancements to deliver a more comprehensive and intuitive search experience for our users," says the senior data engineer.