Last week I spent some time diving more into LLM inference. This is a field I personally find interesting, especially because of the optimizations and engineering behind it. It is a tricky optimization problem where the goal is to make LLM inference possible while controlling hardware costs and keeping high quality.

This is post will also find relevant those who want to run open source LLMs on their own infrastructure. There might be multiple reasons to do that, such as cost control to avoid unpredictable per-token pricing, data privacy to keep sensitive data-in-house, lower latency by serving models closer to where the data lives, and greater flexibility to fine-tune or customize models without depending on a third-party provider's roadmap or rate limits among others.

However, the knowledge required to run those workloads is vast and requires understanding the underlying mode of operation of Large Language Models (LLMs), especially the popular transformer architecture and the SOTA (State of the Art) advances in this area. Let’s dive in.

The planning Phase

First, you have to choose a model for your use case. This is a planning process where you must involve AI engineers, business owners, infra engineers, and DevOps engineers with LLMOps experience. It is very important to know the nuances of choosing different models depending on the business use case.

For the sake of keeping this article short, let’s say you have decided to give it a try with Llama3.1 8B. Before taking any further steps, let’s check what it means by choosing this model.

• It is a multilingual LLM. It means this model was trained on diverse, multilingual datasets. Hence it has inherit the capacity to do translation. The multilingual part is probably not necessary for your use-case.

• This is a text in/text out model. It does not handle images, audios, videos or any other sort of inputs. Only text as input and produce only text as output.

• It uses a transformer architecture. This is very important to know in advance because there are inference servers that support only transformers.

• The model uses Grouped-Query Attention (GQA) for enabling fast inference.

• Context length: 128k. This is also very important though 128k of context length is quite long. Ok it depends. For example, if you are planning to use the LLM as a code agent the context length could exponentially grow. Since in the model card say that it support 2 output modalities: Multilingual text and code. We expect the context length to be big enough for using for coding agents.

• Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

I was able to gather all of the above information from the model card. Now we need to have an idea of what it will take in terms of resources to run this model in production.

The Resource Estimation Phase

One of the first questions you should have: How much memory will it take to store a model with 8B (billion) parameters in GPU memory (VRAM) using FP16 (half precision)?.

The formula to calculate the memory usage for the models weights (parameters) is the following:

It means that for a model with 8 billion of parameters at half precision (FP16) where each parameter occupies 2 bytes we have:

It means that to only store the model’s parameters we required at least 16GB of GCPU VRAM.

In order to have an idea how the VRAM is a bottleneck, we can see in the image below the GPU memory usage distribution when upgrading to a model with 13B of parameters.

We can immediately notice that using same hardware and upgrading our model we are reserving less memory space for KV cache which has an impact in model latency (negative), throughput (negative) and accuracy (probably positive due the increase of numbers of parameters). We will always need to make tradeoffs as shown in the figure below.

For each generated token, caches must stores a key and value vector per transformer layer and head. It is very important that you calculate and have an estimation of the potential KV usage.

One way to get this information without loading the model is checking the config.json in Hugging Face.

We will see that the variables names are slightly different but here are the naming conventions:

n_layers -> num_hidden_layers
n_heads -> num_attention_heads
d_model -> hidden_size
d_head = hidden_size / num_attention_heads

Doing the math, we can come with the result that for the Llama3.1 8B model in FP16 (2 bytes per element), we will need approximately 0,52MB per-token KV cache. For the full 131’072 (128k) context window, the KV cache size is about 68,7 GiB.

That is for Batch size equal to 1. When we increase the batch size, the total memory footprint scales linearly with each additional sequence.

Now we understand that the KV cache prevents us from generating very long sequences and from processing large batches. This is one of the main reasons we need to apply optimizations techniques like quantization (more on this later), KV cache offload, continuous batching and others we will see later.

As you might noticed, if we want to run full context of 128K for the 8B parameter Llama model we are using on this article and using the same hardware (NVIDIA A100 - 40 GB). We cannot run even 1 inference query. We will run in OOM. The only option is upgrade hardware, apply quantization or use a smaller model.

Optimizations

In this section, I present a brief overview of the optimizations most commonly used in LLM inference. Most are already implemented in inference frameworks such as SGLang, vLLM and llm-d.

1-Paged Attention

First published in the now widely cited paper Efficient Memory Management for Large Language Model Serving with PagedAttention.

This is a technique to manage efficiently the KV cache memory of the GPU. It is inspired by the algorithm utilized in classical virtual memory and paging techniques in operating systems.

In earlier systems, KV cache allocation per request or batch reserved memory that went unused. Each allocation reserved space matching the context length limit for the duration of generation, which constrained batch size. PagedAttention improves the memory management, cutting waste to under 4% and increasing throughput by 2-3x.

2-Prefix Caching

This is a technique utilized to avoid redundant prompt computations. The approach is to cache the kv-blocks of processed requests, and reuse these blocks when a new request comes in with the same prefix as previous requests.

As shown in the image below we can see two scenarios when prefix caching is utilized. The first one is when different users utilized the same system prompt (for instance in RAG), this prompt is computed only the first time and then subsequently loaded from cache.

The second example is when a single user is having a multi-turn conversation with the LLM. The session length or context from previous rounds are cached and then subsequently loaded from cache to avoid to re-compute it again.

We can see in figure below how by using prefix-caching we can increase the throughput during inference.

3-Parallelism

There are multiple forms of parallelism within inference systems. This technique allow FLOP operations at different layers to be executed in parallel.

I am not going to go deeper into each on these forms but I will briefly mention them on this article.

Data parallelism: Groups multiple independent requests into batches.

Tensor Parallelism: This is a technique that split layer-wise the weights of transformer model and computations are assigned to independent GPU cores.

Pipeline Parallelism: Mostly used when using big LLMs with multiple GPUs nodes. Basically is a technique to partition the model layer-wise, assigning different layers to separate GPUs for computing.

3D Parallelism: Combined them all

There is also Expert Parallelism for Mixture-of-Experts (MoE) architecture but I am not going to talk about this one on this article.

From vLLM documentation they recommend when using vLLM the following approaches:

• Single GPU (no distributed inference): if the model fits on a single GPU, distributed inference is probably unnecessary. Run inference on that GPU.

• Single-node multi-GPU using tensor parallel inference: if the model is too large for a single GPU but fits on a single node with multiple GPUs, use tensor parallelism. For example, set tensor_parallel_size=4 when using a node with 4 GPUs.

• Multi-node multi-GPU using tensor parallel and pipeline parallel inference: if the model is too large for a single node, combine tensor parallelism with pipeline parallelism. Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. For example, set tensor_parallel_size=8 and pipeline_parallel_size=2 when using 2 nodes with 8 GPUs per node.

4-Quantization

LLM model’s parameters are continuously growing. As shown before, it is required to stores all parameters in memory (GPU VRAM). This means the total model size sets the minimum memory requirement.

Quantization is a method to make models smaller to overcome this issue. Parameters are usually stored in floating-point numbers and inference involves a huge amounts of floating-point operations (FLOPS). There are different floating-point precisions such as FP32, FP16, BF16, INT8, INT4. By lowering the precision we massively reduce the memory usage but at the potential cost of reducing accuracy.

LLM compression improves both throughput and latency and done correctly, it does not significantly degrade the performance of the model.

5-Speculative Decoding

This is a technique to speed up inference. Due the fact that autoregressive generation will generate a single token for every full forward pass. This a sequential process resulting the GPU compute power to stay idle most of the time.

Speculative decoding is technique that combine a target model and a draft model (a more smaller one) in a mechanism where the draft model proposes multiple token for the next generation and the target model verifies those proposals (all of them) in a single forward pass.

Compared with standard autoregressive decoding, which produces on token per pass, this technique reduce the latency during inference and boosting the throughput without any impact on accuracy.

6-Cache Aware Routing

Often the LLM you need to deploy does not fit in a single GPU or even if it does you need multiple GPUs to increase the throughput of the system to be able to serve more users.

In a distributed system when an user send multiple requests during the same session usually you cache them in one server to reduce computing across the whole system.

In distribute inference for LLMs we do something similar. When user is interacting with the LLM asking questions, refining answers, perhaps the LLM is calling tools or sub-agents everything during the same session or context, you don’t want to recompute and cache the context every time and in all machines. This will be a waste of resources.

A technique to solve this is called KV cache aware routing. This mechanism makes intelligent routing decisions based on KV cache awareness.

The way it works is for example the user send a request to the router like this:

curl http://localhost:30080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "prompt": "What is the capital of France?",
    "max_tokens": 100
  }'

Then, send another request with the same prompt prefix:

curl http://localhost:30080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "prompt": "What is the capital of France? And what is its population?",
    "max_tokens": 100
  }'

You should observe that the second request is routed to the same instance as the first request. This is because the KV cache aware router detects that the second request shares a prefix with the first request and routes it to the same instance to maximize KV cache utilization.

Conclusions

As you have seen it is not trivial to run LLMs in your own hardware. There are a lot of tweaks to make and decisions to take depending on your use-case. Also take this article as a basic reference as I am omitting more advanced techniques and I am not going tin depth on the ones mentioned in the article. Thanks for reading!

Sources

• https://horace.io/brrr_intro.html

kipply's blog

Transformer Inference Arithmetic

This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math. The amount of understanding that can be acquired this way is really impressive and practical! A very simple model of latency for inference turns out to be a good fit for emprical results. It's helped me make be…

Read more

5 years ago · 25 likes · 1 comment

• https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices

• https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/

• https://vllm.ai/blog/2023-06-20-vllm

• https://arxiv.org/abs/2309.06180

• https://vllm.ai/blog/2025-09-05-anatomy-of-vllm

• https://vllm.ai/blog/2025-01-27-v1-alpha-release

• https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/

• https://www.aleksagordic.com/blog/vllm#cpt1

• https://docs.vllm.ai/en/stable/serving/parallelism_scaling/

• https://medium.com/@chenhao511132/parallelism-in-llm-inference-c0b6bdc5f693

Wether you are a DevOps Engineer, Platform Engineer, or SRE, you might know Renovate — an automated dependency update tool.

It helps keep your infrastructure up to date without almost manual intervention. In this post, I’ll show how Renovate can be extended to make the most of it. While the examples focus on cloud infrastructure and DevOps, the same approach can be applied by developers who want to keep their code packages and libraries up to date.

How Renovate works?

There are many ways to use it. You can run it as a Github Action, you can self-host your own Renovate server or you can do what majority do which is install the Renovate App in your version control system, let’s say GitHub and then allow Renovate access to the repositories you want to enable it.

Once is installed, you can create a JSON file in your repository with any of the following filenames where you will store the Renovate configuration:

1. renovate.json

2. renovate.json5

3. .github/renovate.json

4. .github/renovate.json5

5. .gitlab/renovate.json

6. .gitlab/renovate.json5

7. .renovaterc

8. .renovaterc.json

9. .renovaterc.json5

10. package.json (within a “renovate” section)

Renovate bot will pickup the configuration and act accordingly.

When it comes to the configuration and starting with Renovate, it is recommended to use one of the presets. These presets contains best practices recommended by Renovate maintainers. One of the most popular presets is “config:recommended” which basically encapsulate the following configuration:

Find more about presets here. I encourage you to get familiar with them.

Another important part of the configuration is the schedule cadence. Scheduling allows you to define when you want to Renovate to scan your repository and create Pull Requests. There are infinite use cases and it depends on each team but the most commons are:

• Run Renovate outside office hours, to free up continuous integration resources for your developers.

• Get updates for certain packages on a regular interval, instead of right away.

• Reduce Renovate bot PR notifications during the day.

You can customize the scheduler with a specific timezone and the schedule cadence. For instance you can have Renovate to run every 2 Tuesdays with the following configuration:

“schedule”: [

“on the 1st and 3rd day instance on Tuesday”

]

This tool is fully customizable. For instance you can configure when to update specific dependencies. Take a look the scheduling documentation.

Below you can see a diagram with the overview of how Renovate is configured and how it works as a Github App.

Subscribe now

Features

Renovate can provide updates for most popular languages, platforms, and registries including: npm, Java, Python, .NET, Scala, Ruby, Go, Docker and more. Supports over 90 different package managers.

Renovate updates code repositories on the following platforms: GitHub, GitLab, Bitbucket, Azure DevOps, AWS Code Commit, Gitea, Forgejo, Gerrit (experimental)

In a nutshell what Renovate does:

• Delivers update PRs directly to your repo

  • Relevant package files are discovered automatically

  • Pull Requests automatically generated in your repo

• Provides useful information to help you decide which updates to accept (age, adoption, pass rates, merge confidence)

• Highly configurable and flexible to fit in with your needs and repository standards

• Largest collection of languages and platforms (listed below)

• Connects with private repositories and package registries

Tips and how to extend Renovate

Finally I would like to provide my 5 cents on how I have extended this great tool and some tweaks that have worked for me.

• Use a central repository to define a configuration for a group of repositories. For instance define all the configurations related to Terraform modules in a central repository. It will save you time and a lot of Pull Requests, believe me, specially when you manage a high number of Terraform modules inside your organization.

• Use automerge when makes sense. When renovate runs it will open a huge list of Pull Requests (PRs). One way to reduce this “spam“ of PRs is to enable automerge for packages with a minor version upgrade or when the change is in lower environments (not production).

• When you manage modules inside your organization and you use Github Releases. Renovate will create PRs on those repositories that consume those modules. You can see how it is fully automated by default 🙂.

• Once Renovate is enabled, you can also access the UI provided by Mend.io to see the runs and have a more detailed view of what it is doing. You just need to go to

  https://developer.mend.io/ and login with your Version Control account, let’s say Github.

• Additionally you can download the CLI and run it locally for troubleshooting, for instance using the —dry-run flag.

renovate --dry-run

Enjoy it 🎃 !

Share

A few weeks ago, I noticed a decrease in the battery performance of my Google Pixel 4a. I didn't have time to investigate further, but while the battery used to last over 24 hours, I now had to recharge my phone just 2 or 3 hours after a full charge. Really?. I knew something was going one, perhaps my phone that I have for more than 3 years was too old and need a replacement 🤔.

But I decided to investigate a bit more and it took me few seconds to realize I was not the only one with that problem recently. My problem was basically in the news!.

Basically what happened is that in January of this year, Google launched the “Pixel 4a Battery Performance Program“.

Google has determined that certain Pixel 4a phones require a software update to improve the stability of their battery’s performance. An automatic update would reduce battery life and charging performance for some "Impacted Devices." Affected customers had one year to choose from three options: sending in their phone for a battery replacement, receiving $50 (or the local equivalent), or getting $100 in Google Store credit toward a new Pixel phone. The support document did not mention any safety or hazard concerns.

I am not going into the details. There are good articles out there expanding the information. I recommend specifically this one who said they reach out directly to Google about this problem and they tried to get answers.

I am just listing Google’s response from the mentioned article:

• It seems to have been built by a Google engineer "on their personal machine, not the proper buildsystem."

• There is no source provided, as would normally be required of a Linux kernel build, though it may only need to be provided on request under the GNU General Public License.

• The maximum charge voltage of certain battery profiles changes from 4.44 volts to 3.95, which would mean batteries cannot charge to anywhere near their former potential.

• There are two main battery profiles, with distinct "ATL" and "LSN" markers; Martin suggests they relate to Amperex Technology Limited and Lishen, manufacturers of battery cells.

• LSN-tagged batteries assigned the "debug" profile can see capacity reduced from 3,080 milliamp hours (mAh) to 1,539 mAh.

To me, this is a fiasco from Google, forcing users to update phones that were working perfectly. We've seen a similar situation with Apple in the past.

Grapehene OS (Open Source) to the Rescue

I didn't give up. After a few hours of research, I decided to replace my phone's operating system with GrapheneOS, and I couldn't be happier! 😃

It is a privacy and security focused mobile OS with Android app compatibility developed as a non-profit open source project.

I will leave some remarks about my transition to Graphene:

• No Google apps or services: If you want and no need any Google apps you can have an Android phone without it. GrapheneOS will never include either Google Play services or another implementation of Google services like microG. Perhaps, it is still possible to install Google Play services and you can install and use any app from the store. On Graphene it is done by a set of fully sandboxed apps without special privileges via our sandboxed Google Play compatibility layer.

• No passive listening services: You won’t have those apps which continuously monitor for wake words and process conversations to offer recommendations

• More battery life: The most important thing is that the problem caused by Google's fiasco is gone, and the battery now performs just as it did before the update. Plus, without any passive listening services, the battery lasts even longer!.

• Bank’ apps: They are a bit problematic to make them run in Graphene. It's likely to work but there's no guarantee. There is a list of Banking Applications Compatibility with GrapheneOS. Check this out if this is very important for you!. I use 3 or 4 bank’s apps and 2 of them didn’t work. Not a big deal for me but worth to mention.

• End Of Life support: For the Google Pixel 4a, Graphene OS does not provided any more firmware or driver security updates, and receive extended support from GrapheneOS via a legacy branch based on Android 13 with only the Android Open Source Project security backports, certain other security patches, and other minimal changes to keep them working. Not a big deal for me, what I have now is way better than before shipped by Google originally on my Pixel. Check out the list of supported devices on Graphene OS.

Conclusions

What started as a frustrating experience turned into an opportunity to take full control of my device. I discovered a better, more secure, and privacy-respecting alternative ♥️. My Pixel 4a now runs smoother and battery lasts longer.

The law has an inevitable limit. You can continue making something smaller and smaller...but only up to a certain point. Then another ways of innovation are required to make more powerful and optimal processors.

Over the years, the demand of more compute resources has been undeniable. With the explosion of what we know today as the internet in the 90's, triggered a surge in the volume of digital data. Major companies like google realize that they could analyze the data using Big data techniques and uncovering insights into user behavior and digital patterns. They have been using this information for optimize their search engines and drive data-driven decision-making.

With a vast amount of data and the nature of machine learning algorithms requiring a huge amount of computing that a general-purpose CPU can not supply. This poses a problem to companies doing Machine Learning and Big Data analysis in the early stages.

In the 1990s, engineers created a specialized compute unit called the GPU (graphics processing unit) to improve video game graphics. Today, GPU cloud servers are used for many computational tasks, including computer clusters, artificial intelligence systems, and deep learning, to enhance the performance of parallel systems. The GPU presented a new paradigm on parallel computing architecture. It was presented as it could run simultaneous execution of two or more parts of a program. This approach combined multiple smaller computers with large memory banks and high-speed processors to form a single, high-performance supercomputer.

GPUs are good at doing lots of simple computations in parallel. This fits perfectly with the workload required for training neural networks like Convolutional Neural Networks (CNN).

Today, there are many of GPU vendors. Moreover companies had created their own custom hardware for optimize AI inference. One example, is the Facebook's hardware platform. The Facebook Zion is comprised of eight CPUs and eight accelerators. The module is disaggregated insofar as the server CPUs are in modular sleds and the accelerators sit in another modular system. The Zion system components are show in the picture below.

This is perfectly fine in cloud environments, where you have the possibility to choose the architecture, capacity and model of CPUs and GPUs achieving a homogeneous stack for training and inference Machine learning pipelines. However, moving ML inference and training to the edge introduces several new challenges.

In the following section, I will discuss these challenges in detail and provide practical recommendations for effective AI deployment at the edge.

AI at the Edge

The main idea of running Machine Learning at the edge is to reduce latency on the output. Moreover, there are cases where the devices deployed in the field does not have a reliable internet connection and inference must happens on the device without depending on the internet. One example of AI at edge is the Iphone feature to unlock your Iphone with your face as an unique identity key.

You don't want the users end up in a situation that they can't get access to their phones because there is not internet connection. Moreover the capture of the image (your face), converted into the inputs for the AI model and the inference must happen really fast.

Engineers have worked to overcome the challenges of enabling AI at the edge. For example, the iPhone's face unlock feature benefits from Apple's AI ecosystem. Since 2017, Apple has included its own mobile GPU, simplifying development for engineers by providing a homogeneous architecture when deploying apps to iPhones.

The Android ecosystem presents a different scenario. There are over 25 mobile chipset vendors which each mixes and matches its own custom-design components. With multiple chip vendors producing CPUs and GPUs for Android devices, the ecosystem is highly fragmented. Engineers aiming to deploy AI at the edge face significant challenges in targeting the various architectures.

Vulkan which is a low-overhead, cross-platform API for high-performance 3D graphics and is a successor to OpenGL and OpenGL ES, aims to provide a layer abstraction to gain control of the stack. Today, early adoption of Vulkan is limited but growing over the years.

This is the challenge with vendor diversity. A holistic approach is to develop ML models capable of running effective inference using only CPUs. This eliminates the need to ensure the model performs well across multiple GPU architectures. However, relying solely on CPUs introduces its own challenges, requiring significant software-level optimizations to make the model more efficient.

Share

Techniques for AI at the Edge

Engineers also face implicit challenges due to the constraints of IoT devices. These devices have limited CPU, GPU, and memory resources, which makes certain techniques necessary to improve AI performance at the edge. The following techniques are recommended:

• Data parallelism: It is common to split the data and send them in parallel to multiple CPUs/GPUs to make the inference faster.

• Model parallelism: Some ML frameworks allow you to split the model having each CPU/GPU core process its own part. This approach is more complex and not all frameworks and hardware vendors allow it.

• Heavy initialization of ML/DL model: The recommendation is to perform initialization once per process/thread and reuse them for running inference.

• Under-utilization: It is common to land in a situation that the tasks handled by the CPU could take 100% of its capacity, while underutilized the GPU.

• Out-of-order outputs: When an order of the outputs are required in a parallelized system, then an extra task is required like indexing and reordering.

• Multiprocessing: Several languages for performing Machine Learning allow you to use multiprocessing techniques to take advantage of parallelism with GPUs. It is worth to mention that not all Deep Learning frameworks supports multiprocessing.

• Training at edge: In some cases, the IoT device may need to learn from data, requiring both ML training and inference to occur on the constrained device. One common approach is to use a model with frozen weights and perform fine-tuning, which is less computationally demanding.

• Optimizations: Model architecture search, weight compression, quantization, algorithmic complexity reduction, increase of training data, refining features sets and micro-architecture specific performance.

The growing importance of deep learning-based applications presents both exciting opportunities and complex design challenges at the edge. TinyML, for instance, is a new approach that simplifies the integration of AI at the edge, enabling applications where sending data to the cloud is impractical. Fortunately, hardware has advanced to the point where real-time analytics are now feasible. For instance, the Arm Cortex-M4 processor can perform more FFTs (Fast Fourier Transform) per second than the Pentium 4, while consuming significantly less power. Similar improvements in power and performance have been made in sensors and wireless communication. TinyML allows us to leverage these hardware advances to create innovative applications that were previously impossible.

To know more about TinyML and AI at the Edge, I recommend the following resources:

• Tiny Machine Learning: Progress and Futures

• Edge Impulse

• Edge Impulse: An MLOps Platform for Tiny Machine Learning

Cosine similarity measures the cosine of the angle between two vectors or, equivalently, the dot product of their normalized forms. It has proven useful in various applications, such as recommender systems and NLP tasks. However it has been proven to have poor results in certain scenarios.

Using cosine similarity as a training objective for machine learning models is valid and mathematically sound. It combines two fundamental operations in deep learning: dot product and normalization. The trouble arises when we push it beyond its intended scope, particularly in cases where the objective function used in model training isn't cosine similarity. Which is usually the case.

It is easy to demonstrate, how sentences that are semantically similar will have poor result when calculating their cosine similarity. For instance let’s have as an example the following 3 sentences:

1. “AI can make you rich.”

2. “AI can make you itch.“

3. “Mastering AI can fill your pockets.“

Let’s calculate the cosine similarity between 1 and 2 and then 1 and 3. What do you expect as a result?. Semantically 1 and 3 are closer so let’s see.

Ah! The vector resulting from sentence (1) is closer to sentence (2) than to sentence (3). That’s likely not the desired outcome in an NLP pipeline or a recommendation system. This simple example demonstrates how cosine similarity can lead to unexpected results.

Cosine similarity is a quick fix for vector comparisons, as shown in the previous example. While it works and can be useful, in some cases—and I would argue in most cases—it masks deeper underlying issues.

So, what can we do about it? There are several alternatives, and it all comes down to one thing: testing, researching, and more testing! Below, I will outline possible solutions, but as you might already know, there’s no silver bullet. It’s all about experimenting and testing with your deep learning model and data.

• Train models directly with respect to cosine similarity.

• Use LLM’s to guide the search, perhaps incorporate an AI agentic approach.

• Use a different metric for comparison as the Euclidean distance or soft cosine similarity.

• Clean and standardize text before embedding before using cosine similarity can help mitigate some of its issues.

Resources

[1] Is Cosine-Similarity of Embeddings Really About Similarity?

The Kubernetes release version 1.30 also named as Uwubernetes (released on April 2024) came with interesting changes and enhancements. Here I am highlighting those I think are the most important:

• Container Resource-Based Pod Autoscaling

• Robust VolumeManager reconstruction after kubelet restart

• Pod Scheduling Readiness

In this article, I won’t go into the details of the Kubernetes 1.30 release. Instead, my focus is to introduce and discuss the significant change in EKS (AWS Managed Kubernetes Service), where Amazon Linux 2023 (AL2023) has replaced Amazon Linux 2 (AL2) as the default operating system for managed node groups.

Note: If you prefer to continue using AL2, you can select it as the AMI type when creating a new node group.

What is Amazon Linux 2023 (AL2023)

From AWS website.

Amazon Linux 2023 (AL2023) is the next generation of Amazon Linux from Amazon Web Services (AWS). With AL2023, you can develop and run cloud and enterprise applications in a secure, stable, and high-performance runtime environment. Also you get an application environment that offers long-term support with access to the latest innovations in Linux. AL2023 is provided at no additional charge.

In general terms, this operating system comes with enhancements for running Linux on the cloud which make it more faster and secured. If you want to know more in details the performance and operational optimizations, you will find useful this link.

There is an extensive documentation as well about comparing AL2 and AL2023, check it out here.

Finally, if you need to know what are the packages and new packages with their version that comes on the new AMI based on AL2023, check this other link here.

There is a lot to say about the transition from AL2 to AL2023, but on this article I am going to focus in a specific change that can affect your Kubernetes workloads when performing this migration.

Containers and AL2023

Before, let me quickly explain what is the Instance Metadata Service (IMDS).

This is a feature that provides metadata about an EC2 instance to the instance itself. This metadata includes details such as the instance ID, AMI ID, security group, and other configuration information.

IMDS also provides temporary credentials for accessing AWS services securely without embedding credentials in the application. Applications running on the instance can access IMDS via HTTP requests to a special local endpoint, 169.254.169.254.

💔 The breaking change

AL2023 requires IMDSv2 by default. IMDSv2 has several benefits that help improve security posture. It uses a session-oriented authentication method that requires the creation of a secret token in a simple HTTP PUT request to start the session. A session's token can be valid for anywhere between 1 second and 6 hours.

For IMDSv2, the default hop count for managed node groups is set to 1. This means that containers won't have access to the node's credentials using IMDS.

Depending how you configured your workloads and in case they need access to the nodes’s credentials using IMDS then you will need to workaround this problem.

Following I show you different alternatives for solving this problem.

Share

🚀 Alternatives to solve this problem

1. Use EKS Pod Identity

   In my opinion, the best way is to configure your workloads in Kubernetes using EKS Pod Identity. Basically you associate an IAM role with a Kubernetes Service Account and then you configure your workloads (pods) to use the service account. More information about it here.

2. Use a Launch template with HttpPutResponseHopLimit

   As mentioned before on this article, for IMDSv2 the the default hop count is set to 1. You can configure this hop limit to any value greater than 1 using a launch template for your EKS managed worker nodes on the Launch Template Metadata Options. For instance:

   {
     "HttpEndpoint" : String,
     "HttpProtocolIpv6" : String,
     "HttpPutResponseHopLimit" : 3,
     "HttpTokens" : String,
     "InstanceMetadataTags" : String
   }

3. Configure instance metadata options for new instances

   Is it possible to set some default EC2 metadata settings at the AWS account level (per region) so any new EC2 instance will take the default settings. Keep in mind that you can override the default settings by specifying the setting at launch time of the EC2 instance.

   Using the AWS CLI you can do this, for instance:

   aws ec2 modify-instance-metadata-defaults \
       --region us-east-1 \
       --http-put-response-hop-limit 2

   Using Terraform you can do it like this:

   resource "aws_ec2_instance_metadata_defaults" "ec2_metadata_defaults" {
     http_put_response_hop_limit = 3
   }

4. Change instance metadata options on existing instances

   In case you need to change the metadata option to an existing running instance. You can do it with the AWS CLI and it won’t required the instance to restart or be replaced at all. You can perform this change for instance like this:

   aws ec2 modify-instance-metadata-options --instance-id i-123456789 --http-put-response-hop-limit 3

Here are the alternatives available to address this situation. You might also be curious about which EC2 instances are still using IMDSv1 and whether they depend on this metadata version. Fortunately, AWS provides comprehensive documentation to guide you through this transition, which you can access here.

Resources

[1] Review release notes for Kubernetes versions on standard support

[2] What is Amazon Linux 2023?

[3] Performance and operational optimizations

[4] Comparing AL2 and AL2023

[5] New Packages in AL2023

[6] Learn how EKS Pod Identity grants pods access to AWS services

[7] EC2 LaunchTemplate MetadataOptions

[8] Configure instance metadata options for new instances

[9] Modify instance metadata options for existing instances

[10] Configure the Instance Metadata Service options

[11] Terraform Resource: aws_ec2_instance_metadata_defaults

[12] Terraform Resource: aws_launch_template

[13] Transition to using Instance Metadata Service Version 2

I will be using Milvus as a vector database and the M3-Embedding model for generating dense and sparse vectors embeddings.

Let’s start with a short description of RAG.

Short description of Retrieval Augmented Generation (RAG)

RAG is a technique that emerged with the great evolution in the last years of Natural Language Generation (NLG) models. It integrates retrieval mechanisms with generative language models to improve output accuracy, effectively addressing the primary limitations of Large Language Models (LLMs).

Traditional NLG models, particularly sequence-to-sequence architectures, generate fluent and coherent text. However, these models depend heavily on training data and often face challenges in producing factually accurate or contextually relevant content for queries requiring knowledge beyond their training data, such as private data.

RAG is an emerging hybrid technique designed to address the limitations of pure generative models integrating two key components:

• A retrieval mechanism, which retrieves relevant documents or information from an external knowledge source.

• A generative module, which processes this information to generate human-like text.

This combination enables RAG architectures to generate fluent text while grounding their outputs in real-world, up-to-date data.

Before I show the implementation of the hybrid search. let’s define the main components of it.

Sparse Vectors

Sparse vectors are used as a data representation in Natural Language Processing (NLP). Their main characteristic is that most of the elements are zero—keeping only what is more relevant in terms of the data representation. This allows sparse vectors to be more accurate when it comes to applications that require precise matching of keywords or sentences.

Common applications:

• Text analysis

• Recommendation systems

• Image processing

Sparse vectors can be generated by for instance using the BM25 method. It relies on the frequency of words in a document and does not attempt to comprehend the meaning of context of the words. It also required the computation of the entire corpus in advance.

Dense Vectors

Dense vectors are a numerical representation of semantic meaning and they are ideal for capturing deep semantic relationships. Compared to sparse vectors, dense vectors encode more information per dimension than sparse vectors, capturing complex patterns and relationships for easier analysis in high-dimensional spaces. For example, in a sparse vector, the vectors for king and queen would be just as dissimilar as the vectors king and apple, even thought king and queen have related meanings.

Common applications:

• Sentiment analysis

• Information retrieval with semantic meaning

• Machine translation

Hybrid Search

Hybrid search leverages the strengths of both approaches, combining the precision of sparse vectors with the deep contextual comprehension of dense vectors. This combination ensures that no important documents are overlooked, whether they align precisely with the query or capture its broader intent.

In the following section I am going to demonstrate how using hybrid search in a RAG pipeline can improve the results of the retrieval phase.

Practical Implementation

To be able to use hybrid search we require a model embeddings with the capability to generate embeddings as dense and sparse vectors. M3-Embedding: A new versatile model for Multi-Linguality, Multi-Functionality, and Multi-Granularity. It provides a uniform support for the semantic retrieval of more than 100 working languages. It can simultaneously accomplish the three common retrieval functionalities: dense retrieval, multi-vector retrieval, and sparse retrieval. Besides, it is also capable of processing inputs of different granularities, spanning from short sentences to long documents of up to 8,192 tokens [4].

For the demonstration I will use the rag-mini-wikipedia dataset which has a corpus of 3200 questions in different topics.

Then I will use Milvus which is an open-source vector database that supports both types of vectors (sparse and dense) in one collection, allowing for hybrid search that enhances the result relevance.

The Jupyter notebook with the implementation and results can be found here and in this article I wanted to highlight the steps performed on the notebook and an analysis of the results. For the whole implementation refer to the Jupyter notebook and the references at the end of the article.

Analysis of the dataset

After downloading the dataset we can explore the content. We can see that it is type Dataset with two features passage and id and it contains 3200 rows.

The query I will explore on the demonstration is the following: Where Elephants live?

Results

After creating the embeddings of the corpus as dense and sparse vectors and inserting them into the Milvus collection I performed 3 types or search: Dense search, Sparse search and Hybrid search and limiting the results to the top 3. Let’s see the results of each of them.

Dense Search Results

distance: 0.6662917137145996

text: Elephants (Elephantidae) are a family in the order Proboscidea in the class Mammalia. They were once classified along with other thick skinned animals in a now invalid order, Pachydermata. There are three living species: the African Bush Elephant, the African Forest Elephant (until recently known collectively as the African Elephant), and the Asian Elephant (also known as the Indian Elephant). Other species have become extinct since the last ice age, which ended about 10,000 years ago, the Mammoth being the most well-known of these.

distance: 0.6637986898422241

text: Elephants are also commonly exhibited in zoos and wild animal parks.

distance: 0.6518104076385498

text: Elephant footprints (tire tracks for scale)Elephants live in a structured social order. The social lives of male and female elephants are very different. The females spend their entire lives in tightly knit family groups made up of mothers, daughters, sisters, and aunts. These groups are led by the eldest female, or matriarch. Adult males, on the other hand, live mostly solitary lives.

Not bad results at all. The documents retrieved using dense search captured the context and semantic of the question and matched with the best documents.

Sparse Search Results

distance: 0.22559085488319397

text: Elephants are mammals, and the largest land animals alive today. The elephant's gestation period is 22 months, the longest of any land animal. At birth it is common for an elephant calf to weigh 120 kilograms (265 lb). An elephant may live as long as 70 years, sometimes longer. The largest elephant ever recorded was shot in Angola in 1956. This male weighed about 12,000 kg (26,400 lb), with a shoulder height of 4.2 m (13.8 ft), a metre (3 ft 4 in) taller than the average male African elephant. The smallest elephants, about the size of a calf or a large pig, were a prehistoric species that lived on the island of Crete during the Pleistocene epoch. Bate, D.M.A. 1907. On Elephant Remains from Crete, with Description of Elephas creticus sp.n. Proc. zool. Soc. London: 238-250.

distance: 0.20308682322502136

text: Elephants (Elephantidae) are a family in the order Proboscidea in the class Mammalia. They were once classified along with other thick skinned animals in a now invalid order, Pachydermata. There are three living species: the African Bush Elephant, the African Forest Elephant (until recently known collectively as the African Elephant), and the Asian Elephant (also known as the Indian Elephant). Other species have become extinct since the last ice age, which ended about 10,000 years ago, the Mammoth being the most well-known of these.

distance: 0.18690812587738037

text: Elephant footprints (tire tracks for scale)Elephants live in a structured social order. The social lives of male and female elephants are very different. The females spend their entire lives in tightly knit family groups made up of mothers, daughters, sisters, and aunts. These groups are led by the eldest female, or matriarch. Adult males, on the other hand, live mostly solitary lives.

We can noticed more poor results using sparse vectors which is expected. If we really want to capture the semantic of the query and document we cannot rely on sparse vectors.

Hybrid Search Results

distance: 1.2508615255355835

text: Elephants (Elephantidae) are a family in the order Proboscidea in the class Mammalia. They were once classified along with other thick skinned animals in a now invalid order, Pachydermata. There are three living species: the African Bush Elephant, the African Forest Elephant (until recently known collectively as the African Elephant), and the Asian Elephant (also known as the Indian Elephant). Other species have become extinct since the last ice age, which ended about 10,000 years ago, the Mammoth being the most well-known of these.

distance: 1.2426867485046387 text: Elephant footprints (tire tracks for scale)Elephants live in a structured social order. The social lives of male and female elephants are very different. The females spend their entire lives in tightly knit family groups made up of mothers, daughters, sisters, and aunts. These groups are led by the eldest female, or matriarch. Adult males, on the other hand, live mostly solitary lives.

distance: 0.6865341663360596

text: Elephants are also commonly exhibited in zoos and wild animal parks.

Very similar results to the dense search but enhanced in the documents retrieved. We can see on this query and any other that you can imagine how hybrid search can improve the results quality during the retrieval phase of a RAG pipeline.

Conclusions

We can constatate that using dense and hybrid search we obtained the best results. For different queries we can obtain different results while having a hybrid search we can enhanced the results during the retrieval phase.

The are different parameters and mechanism we could tweak on this pipeline like the rerank weights for each type of vectors depending in our needs.

Another advanced techniques to explore to enhanced the performance of the RAG pipeline that I do not explore in this article are:

• Creating sub-queries: When a user query is too complicated, we can use an LLM to break it down into simpler sub-queries before passing them on to the vector database and the LLM. Let’s take a look at an example.

• Filtered search: An ANN search identifies vector embeddings similar to a given query but may not always yield accurate results. Adding filtering conditions narrows the search scope to entities matching specific criteria, improving precision.

If you want to see how I am using filtered search to build an AI tool that helps SREs and DevOps teams to gain visibility into their systems and applications check here.

In future posts I would like to explore more advanced RAG techniques and performing observability on RAG using for instance LangSmith and other available tools.

References

[1] A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions

[2] The Milvus Project

[3] Dataset: rag-mini-wikipedia

[4] M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

[5] Jupyter Notebook: Hybrid Search RAG with Milvus

[6] AI tool that helps SREs and DevOps teams to gain visibility into their systems and applications

Constraint programming is a powerful paradigm for solving combinatorial search problems that draws on a wide range of techniques from artificial intelligence, computer science, and operations research. Constraint programming is “programming” partly in the sense of programming in mathematical programming.

The Google OR-Tools are an open source software suite for optimization, tuned for tackling the world's toughest problems in vehicle routing, flows, integer and linear programming, and constraint programming.

In this article, I will demonstrate how to leverage constraint programming and Google's OR-Tools to solve a Sudoku puzzle efficiently.

Solving the Game

The objective of Sudoku is to complete the grid by filling in the digits 1 through 9, ensuring that each digit appears exactly once in every row, every column, and each 3x3 sub-grid.

In the Figure 1, we can see the Sudoku puzzle we will solve. However, as previously explained, by defining a constraint programming model, we can solve any Sudoku puzzle without altering the program's logic. The only modification required is to update the input variables that define the game's initial state.

In constraint programming, models are built using variables and constraints. Let’s define the decision variables.

Decision Variables

We begin by defining integer variables to represent each square in the Sudoku puzzle. These variables are constrained to take on values between 1 and 9.

# p(i,j,n): is true	when the number 'n' is in the cell in the 'ith' row	and	the	'jth' column.

propositions = {}
for i in rows:
    for j in columns:
        propositions[(i, j)] = model.NewIntVar(1, 9, f'i{i}_j{j}')

Constraints

With all the decision variables of our model defined, the next step is to apply the explicit constraints of the Sudoku puzzle. This involves assigning the predefined numbers from the puzzle's initial setup to their corresponding variables.

Set the initial status of the game

# Define explicit constraints (the start status of the given game)
model.Add(propositions[(2, 1)] == 7)
model.Add(propositions[(6, 1)] == 3)
model.Add(propositions[(7, 1)] == 5)
model.Add(propositions[(9, 1)] == 6)

model.Add(propositions[(3, 2)] == 9)
model.Add(propositions[(5, 2)] == 5)
model.Add(propositions[(7, 2)] == 7)

model.Add(propositions[(2, 3)] == 5)
model.Add(propositions[(5, 3)] == 9)

model.Add(propositions[(5, 4)] == 6)
model.Add(propositions[(8, 4)] == 3)
model.Add(propositions[(9, 4)] == 4)

model.Add(propositions[(2, 5)] == 2)
model.Add(propositions[(6, 5)] == 1)
model.Add(propositions[(7, 5)] == 6)

model.Add(propositions[(4, 6)] == 4)

model.Add(propositions[(3, 7)] == 4)
model.Add(propositions[(7, 7)] == 1)

model.Add(propositions[(1, 8)] == 3)
model.Add(propositions[(6, 8)] == 5)

model.Add(propositions[(4, 9)] == 2)
model.Add(propositions[(5, 9)] == 8)
model.Add(propositions[(9, 9)] == 5)

Next, we will define the implicit constraints that stem directly from the core rules of Sudoku.

No digit can occur twice in any of the rows

for i in rows:
  model.AddAllDifferent([propositions[(i, j)] for j in columns])

No digit can occur twice in any of the columns

for j in rows:
    model.AddAllDifferent([propositions[(i, j)] for i in columns])

No digit can occur twice any of the 3x3 sub-grids

for i in range(n):
    for j in range(n):
        model.AddAllDifferent([propositions[(ii, jj)] for ii in range(1 + i * n, 1 + (i + 1) * n) for jj in
                                range(1 + j * n, 1 + (j + 1) * n)])

Solve the game

Constraint programming solves problems by combining systematic search with inference techniques. The search explores all possible variable-value combinations, forming a tree where the root represents the original problem.

Here the CPSolver from Google OR-Tools is invoked with the given task to solve the problem and provide the solution of our Sudoku puzzle.

solver = cp_model.CpSolver()
solver.SearchForAllSolutions(model, SolutionPrinter(propositions))

Solutions

Finally, with the help of the SolutionPrinter python class we can print the all solutions found by the solver.

solution 1
[2, 6, 1, 8, 4, 7, 5, 3, 9]
[7, 4, 5, 9, 2, 3, 6, 8, 1]
[8, 9, 3, 1, 5, 6, 4, 2, 7]
[1, 8, 7, 5, 9, 4, 3, 6, 2]
[4, 5, 9, 6, 3, 2, 7, 1, 8]
[3, 2, 6, 7, 1, 8, 9, 5, 4]
[5, 7, 8, 2, 6, 9, 1, 4, 3]
[9, 1, 4, 3, 8, 5, 2, 7, 6]
[6, 3, 2, 4, 7, 1, 8, 9, 5]

solution 2
[2, 6, 8, 7, 4, 9, 5, 3, 1]
[7, 4, 5, 1, 2, 3, 6, 8, 9]
[1, 9, 3, 5, 8, 6, 4, 2, 7]
[8, 1, 7, 9, 5, 4, 3, 6, 2]
[4, 5, 9, 6, 3, 2, 7, 1, 8]
[3, 2, 6, 8, 1, 7, 9, 5, 4]
[5, 7, 4, 2, 6, 8, 1, 9, 3]
[9, 8, 1, 3, 7, 5, 2, 4, 6]
[6, 3, 2, 4, 9, 1, 8, 7, 5]

solution 3
[2, 6, 1, 7, 4, 8, 5, 3, 9]
[7, 4, 5, 9, 2, 3, 6, 8, 1]
[8, 9, 3, 1, 5, 6, 4, 2, 7]
[1, 8, 7, 5, 9, 4, 3, 6, 2]
[4, 5, 9, 6, 3, 2, 7, 1, 8]
[3, 2, 6, 8, 1, 7, 9, 5, 4]
[5, 7, 8, 2, 6, 9, 1, 4, 3]
[9, 1, 4, 3, 8, 5, 2, 7, 6]
[6, 3, 2, 4, 7, 1, 8, 9, 5]

solution 4
[2, 6, 1, 9, 4, 8, 5, 3, 7]
[7, 4, 5, 1, 2, 3, 6, 8, 9]
[8, 9, 3, 7, 5, 6, 4, 2, 1]
[1, 8, 7, 5, 9, 4, 3, 6, 2]
[4, 5, 9, 6, 3, 2, 7, 1, 8]
[3, 2, 6, 8, 1, 7, 9, 5, 4]
[5, 7, 8, 2, 6, 9, 1, 4, 3]
[9, 1, 4, 3, 8, 5, 2, 7, 6]
[6, 3, 2, 4, 7, 1, 8, 9, 5]

solution 5
[2, 8, 6, 7, 4, 9, 5, 3, 1]
[7, 4, 5, 1, 2, 3, 6, 8, 9]
[1, 9, 3, 5, 8, 6, 4, 2, 7]
[8, 1, 7, 9, 5, 4, 3, 6, 2]
[4, 5, 9, 6, 3, 2, 7, 1, 8]
[3, 6, 2, 8, 1, 7, 9, 5, 4]
[5, 7, 4, 2, 6, 8, 1, 9, 3]
[9, 2, 1, 3, 7, 5, 8, 4, 6]
[6, 3, 8, 4, 9, 1, 2, 7, 5]

The full code is provided below:

from ortools.sat.python import cp_model

rows = [1, 2, 3, 4, 5, 6, 7, 8, 9]
columns = [1, 2, 3, 4, 5, 6, 7, 8, 9]
n = 3  # size of sub-grids


class SolutionPrinter(cp_model.CpSolverSolutionCallback):
    def __init__(self, propositions):
        cp_model.CpSolverSolutionCallback.__init__(self)
        self.propositions_ = propositions
        self.solutions_ = 0

    def OnSolutionCallback(self):
        self.solutions_ = self.solutions_ + 1
        print("solution", self.solutions_)

        for i in rows:
            row_solution = []
            for j in columns:
                row_solution.append(self.Value(self.propositions_[(i, j)]))
            print(row_solution)

        print()


def main():
    # Create the CP-SAT model.
    model = cp_model.CpModel()

    # Create the decision variables
    # p(i,j,n): is true	when the number 'n' is in the cell in the 'ith'	row	and	the	'jth' column.
    propositions = {}
    for i in rows:
        for j in columns:
            propositions[(i, j)] = model.NewIntVar(1, 9, f'i{i}_j{j}')

    # Define explicit constraints (the start status of the given game)
    model.Add(propositions[(2, 1)] == 7)
    model.Add(propositions[(6, 1)] == 3)
    model.Add(propositions[(7, 1)] == 5)
    model.Add(propositions[(9, 1)] == 6)

    model.Add(propositions[(3, 2)] == 9)
    model.Add(propositions[(5, 2)] == 5)
    model.Add(propositions[(7, 2)] == 7)

    model.Add(propositions[(2, 3)] == 5)
    model.Add(propositions[(5, 3)] == 9)

    model.Add(propositions[(5, 4)] == 6)
    model.Add(propositions[(8, 4)] == 3)
    model.Add(propositions[(9, 4)] == 4)

    model.Add(propositions[(2, 5)] == 2)
    model.Add(propositions[(6, 5)] == 1)
    model.Add(propositions[(7, 5)] == 6)

    model.Add(propositions[(4, 6)] == 4)

    model.Add(propositions[(3, 7)] == 4)
    model.Add(propositions[(7, 7)] == 1)

    model.Add(propositions[(1, 8)] == 3)
    model.Add(propositions[(6, 8)] == 5)

    model.Add(propositions[(4, 9)] == 2)
    model.Add(propositions[(5, 9)] == 8)
    model.Add(propositions[(9, 9)] == 5)

    # B) Constraints: no digit can occur twice in any of the rows or columns
    # or in any of the 3x3 sub-grids

    # no digit can occur twice in any of the rows
    for i in rows:
        model.AddAllDifferent([propositions[(i, j)] for j in columns])

    # no digit can occur twice in any of the columns
    for j in rows:
        model.AddAllDifferent([propositions[(i, j)] for i in columns])

    # no digit can occur twice any of the 3x3 sub-grids
    for i in range(n):
        for j in range(n):
            model.AddAllDifferent([propositions[(ii, jj)] for ii in range(1 + i * n, 1 + (i + 1) * n) for jj in
                                   range(1 + j * n, 1 + (j + 1) * n)])

    # Declare the solver and search for all solutions and print them in the callback Class
    solver = cp_model.CpSolver()
    solver.SearchForAllSolutions(model, SolutionPrinter(propositions))


main()

Getting the some statistics on the solution found we can see interesting things as the walltime, usertime, deterministic_time, branches, propagations and of course the status.

Response stats: CpSolverResponse summary:
status: OPTIMAL
objective: 0
best_bound: 0
integers: 6
booleans: 8
conflicts: 0
branches: 100
propagations: 239
integer_propagations: 419
restarts: 49
lp_iterations: 0
walltime: 0.010958
usertime: 0.010958
deterministic_time: 0.00225428
gap_integral: 0
solution_fingerprint: 0x1ad791e213b2d87f

More information about these metrics can be found here.

Constraint Programming represents one if the closest approaches computer science has yet made to the Holy Grail of programming: the user states the problem, the computer solves it.

Eugene C. Freuder, Inaugural issue of the Constraints Journal, 1997

Share

References

[1] Francesca Rossi, Peter van Beek, and Toby Walsh. 2006. Handbook of Constraint Programming. Elsevier Science Inc., USA.

[2] Google OR-Tools

Leave a comment

The letter can be found here, but it is crucial to delve deeper into their warnings and understand how these generative AI applications can harm society and potentially lead to catastrophic consequences. Understanding these risks is vital for addressing the ethical and societal implications of AI development. Specifically in Large Language Models.

Environmental Risks and Equality

Recent advancements in hardware and techniques for training neural networks have led to the development of a new generation of large models trained on vast datasets. These models have achieved significant improvements in accuracy across numerous natural language processing (NLP) tasks. However, these gains are heavily dependent on the availability of extensive computational resources, which require substantial energy consumption.

As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware.

Recent benchmarks have quantified the costs of model training and development in both dollars and estimated CO2 emissions. While the average person is responsible for approximately 5 tons of CO2 annually, the authors found that training a large Transformer model using neural architecture search emitted an estimated 284 tons of CO2. Furthermore, training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to consume as much energy as a trans-American flight.

The community calls on researchers and companies to report training time and computational resources required when proposing models intended for re-training on downstream tasks, such as new domain adaptation or task-specific fine-tuning. This transparency will facilitate comparisons across models, enabling subsequent users to accurately determine whether the required computational resources are feasible for their specific settings.

Equal Access to Computational Resources

Recent advances in computing come at a high price, making them inaccessible to many who seek access.

Restricting this type of research to industry labs disadvantages the NLP research community in several ways. Researchers with innovative ideas but without access to large-scale computing resources are unable to pursue their concepts. This restriction limits certain types of research based on financial resources, further entrenching the "rich get richer" cycle in research funding.

Successful and well-funded groups continue to attract more funding due to their existing achievements. Additionally, the high start-up costs of building in-house resources compel less resource-rich groups to depend on cloud computing services like AWS, Google Cloud, and Microsoft Azure.

Prioritize Hardware and Algorithms that are Computationally Efficient.

It is recommended a collaborative effort between industry and academia to promote the development of more computationally efficient algorithms and energy-efficient hardware. Additionally, efforts should be made to optimize software for better resource efficiency.

Biases in Large Internet Datasets

The Internet, with its vastness and diversity, might suggest that large datasets like Common Crawl (which includes petabytes of data collected over 8 years of web crawling, a filtered version of which is used in GPT-3 training) are broadly representative of diverse viewpoints. However, closer examination reveals factors that limit participation, discussion inclusion based on crawling methods, and filtered content. Consequently, voices aligned with dominant perspectives, including white supremacist and misogynistic views prevalent in US and UK English, are disproportionately represented. This overrepresentation not only exceeds their prevalence in the general population but also exacerbates biases and harms when these datasets are used to train models.

The contributors to Internet text collections are disproportionately younger and predominantly from developed countries due to uneven global Internet access. Specific subsets, such as GPT-2's data sourced from Reddit outbound links, reflect demographics where a majority are young men in the United States. Similarly, surveys of Wikipedia contributors indicate a low representation of women, highlighting skewed participation demographics in these datasets.

Social movements introduce new norms, languages, and modes of communication. This poses challenges for language models (LMs), as reliance on these models risks perpetuating outdated and less inclusive viewpoints—a phenomenon known as 'value-lock'.

For example, the Black Lives Matter (BLM) movement has influenced the creation and editing of Wikipedia articles. As the movement gained momentum, articles on shootings of Black individuals increased in frequency and were generated more quickly. Notably, past incidents of police brutality and shootings were updated alongside new events, demonstrating how social movements connect events over time to construct coherent narratives.

Recent studies confirm that large language models (LMs) exhibit biases such as stereotypical associations and negative sentiment toward specific groups. Additionally, there is evidence of intersectional bias, where models like BERT, ELMo, GPT, and GPT-2 encode more bias against identities marginalized along multiple dimensions than expected. These biases are attributed to characteristics of the training data. For example, recent research reveals that BERT associates phrases about disabilities with negative sentiments, and topics like gun violence and homelessness are disproportionately linked to discussions of mental illness. Similarly, GPT-3, trained on extensive data primarily from Common Crawl, can generate highly toxic sentences even when prompted with non-toxic inputs. Their analysis of GPT-2's training data also uncovered significant content from unreliable news sites and banned subreddits.

Share

Synthetic Text as Meaningful Source

Traditional n-gram language models are limited to modeling local dependencies, predicting each word based on a small sequence of preceding words. In contrast, Transformer-based language models like GPT-3 and GPT-4 can capture much larger contexts, generating text that is not only fluent but also coherent across paragraphs.

For instance, to demonstrate the GPT-4’s capability by prompting it with a specific text excerpt, after which it autonomously generated coherent paragraphs, including answers in a Q&A format:

Question: What is the name of the Russian mercenary group?

Answer: The Russian mercenary group often referred to in the media is known as the Wagner Group. This private military company (PMC) has gained attention for its involvement in various conflicts, including in Syria and Ukraine, where it has operated in support of Russian interests. The Wagner Group is known for its operations alongside Russian military forces and has been linked to controversial activities and alleged human rights abuses.

Question: Where is the Wagner group?

Answer: The Wagner Group, a Russian private military company (PMC), has been known to operate in several conflict zones and areas of interest to Russia. Some of the places where Wagner Group operatives have been reported include: Syria, Ukraine, Libya, Central African Republic (CAR) and Sudan.

Question: Who is the leader of the Wagner group?

Answer: The leader of the Wagner Group, the Russian private military company (PMC), is believed to be Dmitry Utkin. Dmitry Utkin is a former Russian military officer who reportedly founded the Wagner Group and has been linked to its operations in various conflict zones. Utkin, also known by his call sign "Wagner," is said to have a background in military intelligence and has been associated with coordinating and commanding Wagner Group operations. However, due to the secretive nature of private military companies like Wagner, detailed information about its leadership and structure can be difficult to verify.

This example showcases GPT-4's ability to maintain coherence and relevance to the initial prompt, exemplifying its broader contextual understanding compared to traditional models.

Text produced by a Large Language Model lacks grounding in communicative intent, any model of the world, or any model of the reader's state of mind. This is inherent because the training data never involved sharing thoughts with a listener, nor does the machine possess the capability to engage in such interaction. Despite the increasingly fluent nature of automatically generated text, our perception of natural language remains influenced by our own linguistic competence and inclination to interpret communicative acts as conveying coherent meaning and intent, whether or not they actually do. The crux of the issue lies in the fact that if one side of the communication lacks genuine meaning, any comprehension of implicit meaning is illusory and stems from our unique human comprehension of language, which exists independently of the model.

I hope this article provides you with a comprehensive understanding of the risks and issues associated with Large Language Models. It's crucial for consumers of these models to be aware of these considerations. Artificial Intelligence, including LLMs, should not remain an abstract and complex field for the general public. It's essential to explain in straightforward terms the challenges researchers currently face and the societal hurdles ahead of us.

Subscribe now

Denoising autoencoders exhibit versatility across various domains or application areas, often requiring slight adjustments based on the nature of the dataset being utilized. In this tutorial, we will explore one such variant known as convolutional denoising autoencoders, specifically designed for denoising image data.

These models take a noisy image as input and employ multiple convolutional operations to extract crucial features. These latent features are then fed into a series of deconvolutional layers to reconstruct a cleaner version of the image, maintaining the same height and width.

Implementation of an Autoencoder

The implementation of a basic AE consisted on a network architecture densely connected layers as follow:

• Layer 1: 128 Neurons with ReLU activations

• Layer 2: 64 Neurons with ReLU activations

• Layer 3: 32 Neurons with ReLU activations

• Layer 4: 64 Neurons with ReLU activations

• Layer 5: 128 Neurons with ReLU activations

• Layer 6: 784 Neurons with Sigmoid activation

On this AE, I’ve use the Rectified Linear Unit (ReLU) activation functions for the input and hidden layers and a Sigmoid function as the output layer of the Artificial Neural Network (ANN).

The ReLU function returns 0 if it receives any negative input, but for any positive value x, it returns the value back. In practice it is fast to compute, hence it has become the most used activation function on ANNs.

The sigmoid function is a special form of the logistic function. This function is used as an output layer when you want to guarantee that the predictions will always fall within a given range of values, 0 to 1.

Weights Initializer

During the process of constructing the AE, the TensorFlow initializer RandomNormal was used on the initial experiments. But it was noticed that after the first epoch, the output matrix of the sigmoid layer was containing NaN values. This occurs due a saturation of the sigmoid function. A typical solution for this is problem is use another initialiser that solves this problem or implement a sigmoid function that is numerical stable. On my case I resulting using the HeNormal initialiser provided by TensorFlow.

Optimizer

The Adaptive moment estimation (Adam), is an adaptive learning rate algorithm for first-order gradient-based optimisation. It is computationally efficient and has a little memory footprint.

This was the optimizer utilized to update the weights for the AE during training phase. It requires less tuning of the learning rate hyperparameter. The default learning rate η=0.001 has been used.

Here is an illustrative diagram presenting the key components involved on the training phase of an Artificial Neural Network.

Results

The code that implement this AutoEncoder using the popular MNIST dataset can be found here. On this case we wanted to show how to implement a Class using Keras.

The main goal of the training gradient-based phase is to reduce the loss. The loss values across 200 epochs is shown in the following image.

It is noticed how the model was making improvements on each iteration. It was noticed that after around epoch 70-80 the model was not making any more significant improvement. The final loss obtained after 200 epochs was 0.0479.

In order to evaluate the performance of the model in a visual manner. A group of 3 randoms data points (images) were selected from the test dataset and a visualisation of the original, noisy and reconstructed image were generated.

It's evident that the model demonstrates good performance, effectively reconstructing a nearly identical image from a noisy input.

With S3 Express One Zone, you have single digit millisecond access, increasing access speed up to 10x, while reducing costs by 50% compared to standard S3. Your data is redundantly stored on multiple devices within a single Availability Zone (AZ) with 99,95% of availability within a single AZ.

Until now, we had the possibility of choosing in which AWS region our S3 standard bucket would be located. With S3 Express One Zone you must select a specific AWS Availability Zone where the bucket will be created and where all your data will be saved. Data is stored in a different bucket type—an S3 directory bucket—which supports hundreds of thousands of requests per second. This new bucket type has a hierarchical namespace and stores object key names in a directory-like manner, as opposed to the flat key structure of traditional S3 buckets.

S3 Express introduces a new session-based authorization capability that reduces the latency associated with S3 request authorizations. This new capability can be used to create and periodically refresh your connection sessions to the new bucket type.

Let's first refresh how AWS manages the Availability Zones (AZs) in their infrastructure.

As today, S3 Express One Zone is supported in the following Regions and Availability Zones.

Subscribe now

Benchmarking

I have decided to test this new bucket storage class and compare it with the standard one. The idea is to compare how different in terms of speed file transfers can be made.

To carry out the corresponding tests I have created two types of buckets using Terraform. A standard bucket and a directory bucket (Express One Zone).

For transferring the files, I’ve used the AWS CLI with the S3 Sync command to send the files from the EC2 instance to the two buckets.

The code in terraform is quite simple and you can see it below.

• The code to create an S3 Express One Zone bucket:

resource "aws_s3_directory_bucket" "example_express_bucket" {
  bucket = "example-express-bucket--use1-az4--x-s3"

  location {
    name = "use1-az4"
  }

  force_destroy = true
}

Keep in mind that for this type of bucket, the name must be in the format: [bucket_name]--[azid]--x-s3. Where azid is the Availability Zone ID.

• The code to create an S3 Standard bucket:

resource "aws_s3_bucket" "example_clasical_bucket" {
  bucket = "example-clasical-bucket"
  force_destroy = true
}

Buckets where created successfully, we can see them on AWS GUI.

In order to test the bandwidth speed, we have to send and/or retrieve files to the bucket from an EC2 instance. For that I’ve spin up an EC2 instance and wrote a small Shell script to create random files with specific sizes for our benchmark tests.

The Shell script must be executed passing two parameters. One is the number of files you want to generate (file_numbers) and the second one is the size for each of those files (file_size_in_bytes).

Example, to generate 100 files of 100k bytes, you execute ./random.sh 100 100k

You can see the script below.

#!/bin/bash

# Check if the correct number of arguments is provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <file_numbers> <file_size_in_bytes>"
    echo "Example: $0 100 200k"
    exit 1
fi

file_numbers=$1
file_size=$2

for i in $(seq 1 $file_numbers)
    do
        # Generate random data and write it to the file
        dd if=/dev/urandom of=files/$i.txt bs=$file_size count=1 status=progress
    done

Experiments

Now, we've reached the funny part of the game. I've executed two experiments related to transmitting files from the EC2 instance to AWS S3, focusing on the assessment of writing operations.

For the experiment 1, I generate 1000 files each of 1 Mb with this command: ./random.sh 1000 1M

For experiment 2 , 50000 files were generated each of 100kb size with the following command: ./random.sh 50000 100k

We can see the files were created on the EC2 instance and are ready to be uploaded to AWS S3.

In the case of experiment 1, we can see the files’s size:

We can confirmed files were uploaded and we can see them on AWS GUI.

When uploading the files to S3, I measured the time it took on each experiment and the average bandwidth of the transferring operation.

Results

Here I present the results and finding during the experiments. On the table below we can see the results for each experiment.

We can notice that there are no big difference on first experiment. Having 17 seconds v.s. 15 seconds is not a huge difference and this is because we sent 1000 files of 1Mb and the new storage class (S3 Express One Zone) provides better performance when transferring smaller objects. We can confirm this on second experiment when sending more files and more smaller than on experiment 1 and we gain the double of time for the transaction time.

The reason is because latency usually impacts small files much more than larger files as we can constate in our experiments.

But, I wasn’t completely happy with the results. The docummentation mentioned that we can obtain up to 10x speed but I only got 2x. After further investigating I realize that we can tune the AWS Sync command to improve performance by modifying the value of max_concurrent_requests. This value sets the number of requests that you can send to Amazon S3 at a time. The default value is 10, but you can increase it to a higher value.

Then I re-ran the second experiment and obtained better results.

We gain 5x on performance!. That’s great.

To conclude:

• The new S3 Express One Zone storage class (a.k.a Directory Bucket) better perform on large amount of smaller files.

• AWS S3 batch operations will get more beneficial of it.

• I’ve performed the experiments with AWS CLI, but if you have an application that use, for instance the AWS SDK and can perform batch operations in parallel, then this new storage class is a good candidate to gain performance.

• To gain the most beneficial of it, you must set your workload on the same Availability zone as your Directory bucket is located.

• Be aware that the Directory bucket structure is different than the traditional S3 buckets.

Share