04 Oct 2026
What is the scariest thing you can hear at work on a Friday afternoon?
For me it is the question:
How did we get these numbers?
Especially when followed by some mumbling
about an uncommitted shell script on someone’s laptop
that produced a text file. You know, the text file that was
copied into a spreadsheet to generate a plot that was then pasted
into Word.
Oh, and by the way, the script is gone now. But don’t worry, so are the input
files. And I might have tweaked the formulas before (or was it after?) generating the plot.
I really wanted to try something out!
Sorry, what was the question again?
This is a story about my struggle with this question over the years.
It’s also a story about defining complex problems and what that means
for solving them.
This story follows my journey towards benchmarking provenance, from
scattered spreadsheets and tar files to automated, systematic,
and reproducible benchmarking.
It explains how today, at my current job at ZeroPoint, anyone can run
complex, reproducible benchmarks of our hardware at the press of a button.
Do you also hate stressful questions on Friday afternoons? If yes, read on!
New product, new me
It all started with a new product. Maybe it always does. At ZeroPoint, we
were developing a compression hardware IP for expanding the capacity of CXL memory
and we were soon going to evaluate it.
Since CXL memory is about exposing a new memory tier,
we wanted to run real workloads in a full system and measure how their performance
was affected by our compression hardware.
However, it quickly became clear that the
evaluation dimensions were almost endless. You have the configuration of the
hardware IP with tens of parameters like version, compression algorithm, tuning
parameters, etc. Then there’s the firmware with its own set of parameters.
And we haven’t
even started considering the host system setup, the input files used by the
workloads, the build configurations and versions of the workloads, the kernel configuration,
the CPU frequency, etc. You get the idea.
Of course, one way to go through life is to ignore problems until they hit you in
the face. For this particular problem, ignoring it could mean just using
spreadsheets. At first glance, they don’t sound so bad, right?
Whip up a script that outputs measurements in CSVs, then plug those into
a spreadsheet and I can get you the first plot by lunch. And if your spreadsheets
are in the cloud, you can easily share the numbers with anyone. Not
a bad pace, right? Well WRONG.
If you take away one thing from this article, let it be that
spreadsheets are one of the biggest traps in benchmarking.
The trap is simple: spreadsheets make it very easy to skip any kind of provenance.
By provenance, I mean information about
how some specific measurement came to be.
Such information allows you to:
- reproduce the measurement and get the same result
- if a measurement is wrong, understand why that might be the case.
Spreadsheets are bad for provenance because they are out-of-band. You run
your experiment in place A and then you copy the results into a spreadsheet in place B.
Even if you are diligent and you include some manual provenance,
e.g., the file path of your input
data in the spreadsheet, there’s no guarantee that someone won’t edit
that file tomorrow. More importantly, nobody can use that spreadsheet alone
to rerun the experiment and get the same results.
With this benchmarking conundrum at hand, I was called in to find a better
alternative. The problem was that I was unsure about what the problem was, or
what a good solution would look like. Thankfully, I wasn’t starting from scratch.
I struggled with benchmarking provenance a lot during my PhD and I never found a
fully satisfactory solution, so it was kind of a nemesis that came back to haunt me.
The ghost of Christmas provenance.
Me, myself, and my provenance
When you do a PhD, you have to do a lot of experiments. If you’re
good and a bit lucky, you get some nice results that you can use
to write a publication. Then you submit this publication and you wait a
few months for peer review. When the reviews come, you usually need to go
back to your results and use them to answer reviewer questions.
That involves the raw data you used to make the figures in the paper.
In some (unfortunate!) cases, the reviewers might ask you for more experiments.
If your experiments are not reproducible, this
process can quickly turn into a nightmare.
So I had a method. It was not great and it felt like a house of cards but it
served me well for five years without any hiccups, so I’ll tell you all
about it. Or, even better, let’s design it from scratch together.
In the simplest possible terms, an experiment is some code, some binary artifacts, and some
measurements, right?
- What system can keep track of code? Git.
- What system can keep track of measurements? Spreadsheets.
- What system can keep track of binary artifacts? Many. Let’s say tar files.
Great! We have our system. Commit the code. Run the
experiment. Pack everything into a tar archive. Put the measurements, the
path to the tar file and the commit hash of the code into a spreadsheet. If you’re
feeling fancy, you can also record the IDs of the relevant paper figures
in another column. For example:
| Commit |
Date |
Raw Data |
Rate |
Latency |
Figures |
| 14f3d87 |
2026-10-03 |
~/paperA/exp.tar |
30 GB/s |
990 ms |
1, 3 |
BOOM. We have provenance!
So why not use something like that now at ZeroPoint? It obviously worked for me before and
required very little effort to set up. What could go wrong?
Social experiments
Spoiler alert: what could go wrong is other people.
A whole team would now be involved in the benchmarking of the product.
How long do you think it takes before the spreadsheet starts falling apart?
Before people forget to add data, or add some wrong data, or overwrite
other people’s data?
I am a practical person. I considered the option of moving to a remote island
and avoiding other people and the pesky problems they cause. But I persevered.
And I started realizing that I had just stumbled upon the most important
aspect of the problem: control.
More precisely, the lack of control that people should have over the experiments.
People make mistakes. When those mistakes pollute the provenance itself, it
can be impossible to go back. So let’s not allow them to.
- How do you remove control during experiment runs?
Automate everything.
- How do you prevent someone changing the data of the experiments later?
Make the experiments immutable.
- How do you prevent someone overlooking some dimensions of the experiment?
Make the process systematic.
That sounds promising. Spreadsheets did have one advantage, though.
They were easy to share both with
technical and non-technical people. It would be very nice to keep that property
in the new solution.
To summarize, I wanted automated, immutable, reproducible experiments with
systematic tracking of their dimensions. And easily shareable too. Shouldn’t
be hard to do, right?
Version my data, please
And indeed, it wasn’t. Having defined the problem clearly, the solution
came almost organically. So organically that I don’t really remember
exactly how. With the requirements written down, it would not have been difficult
to create the infrastructure myself. But I
was lucky enough to have someone else do my work for me. In my book, that’s the best kind
of solution!
More specifically, the solution I chose came in the form of a framework called
DVC, which stands for Data
Version Control. Even though it is marketed for AI/ML and data science, it
allowed me to model the benchmarking problem and gave me a very elegant base for
a full provenance solution. DVC defines what an experiment is and models its
provenance in a systematic way. All that remained was to orchestrate
the automated experiment execution and result collection.
In the end, DVC augmented with my own infrastructure made hardware benchmarking a
single-command procedure.
Run your experiment and its data is stored immutably in an artifact store,
while its measurements are pushed to a database and visualized in a dashboard.
Nowhere in the process does a (non-malicious) user have the chance to leave out any
part of the experiment configuration or change the results. What you see is what you get.
I cannot overstate the peace of mind that such infrastructure gives to a team.
Being able to prove which configuration was used to produce a specific measurement
and what post-processing was applied afterward can dramatically increase the confidence
in the results. This, in turn, frees up time for things that matter:
more efficient execution, more experimentation, and more investigation of the
performance of the product itself.
Solving problems by defining problems
Reflecting on this journey, I come back to something that I have been
considering a lot over the years. The more experience I gain, the more it seems that
most of my job is about clearly defining the problem I am trying to solve.
How I arrive at that definition varies, depending on the type of problem.
In this case, it helped to reflect on what I had tried before and why it couldn’t work now.
By the way, such reflection also works great at the team level. Ask “why can’t [the simplest
solution] work in our case?” and then try (really try!) to answer this question honestly.
You might be surpised with the answers you end up with.
When I work on R&D projects, I tend to define the problem by
trying to solve it. Sounds counterintuitive, but the point is not to
succeed. The first attempt usually fails,
but the failure itself usually reveals aspects of the problem
not considered before. I feel that this approach works better in
R&D projects because there are too many “unknown unknowns” for
reflection to give any useful results.
Whatever technique you use, I think that the skill of clearly defining problems is
more important than ever. Nowadays, AI can solve a lot of engineering problems.
With the correct guidance, it can help you solve some really hard ones too.
But in my experience, if you don’t define your problems clearly, AI is not much help.
Without a good problem definition, AI can even make things worse, because it deprives
you of the step-by-step problem-solving process that could have revealed
a wrong problem definition.
When AI inevitably gives you an intricate solution for a different problem than the one you had,
you have wasted time, money, and energy that could be better spent elsewhere.
So what are you still doing here? Go out there and find some problems to define!
03 Oct 2026
What is the scariest thing you can hear at work on a Friday afternoon?
For me, it is the question:
How did we get these numbers?
Especially when followed by some mumbling
about an uncommitted shell script on someone’s laptop
that produced a text file. You know, the text file that was
copied into a spreadsheet to generate a plot that was then pasted
into Word.
Oh, and by the way, the script is gone now. But don’t worry, so are the input
files. And I might have tweaked the formulas before (or was it after?) generating the plot.
I really wanted to try something out!
Sorry, what was the question again?
This is a story about my struggle with this question over the years.
It’s also a story about defining complex problems and what that means
for solving them.
This story follows my journey towards benchmarking provenance, from
scattered spreadsheets and tar files to automated, systematic,
and reproducible benchmarking.
It explains how today, at my current job at ZeroPoint, anyone can run
complex, reproducible benchmarks of our hardware at the press of a button.
Do you also hate stressful questions on Friday afternoons? If so, read on!
New product, new me
It all started with a new product. Maybe it always does. At ZeroPoint, we
were developing a compression hardware IP for expanding the capacity of CXL memory
and we were soon going to evaluate it.
Since CXL memory is about exposing a new memory tier,
we wanted to run real workloads in a full system and measure how their performance
was affected by our compression hardware.
However, it quickly became clear that the
evaluation dimensions were almost endless. You have the configuration of the
hardware IP with tens of parameters like version, compression algorithm, tuning
parameters, etc. Then there’s the firmware with its own set of parameters.
And we haven’t
even started considering the host system setup, the input files used by the
workloads, the build configurations and versions of the workloads, the kernel configuration,
the CPU frequency, etc. You get the idea.
Of course, one way to go through life is to ignore problems until they hit you in
the face. For this particular problem, ignoring it could mean just using
spreadsheets. At first glance, they don’t sound so bad, right?
Whip up a script that outputs measurements in CSVs, then plug those into
a spreadsheet and I can get you the first plot by lunch. And if your spreadsheets
are in the cloud, you can easily share the numbers with anyone. Not
a bad pace, right? Well WRONG.
If you take away one thing from this article, let it be that
spreadsheets are one of the biggest traps in benchmarking.
The trap is simple: spreadsheets make it very easy to skip any kind of provenance.
By provenance, I mean information about
how some specific measurement came to be.
Such information allows you to:
- reproduce the measurement and get the same result
- if a measurement is wrong, understand why that might be the case.
Spreadsheets are bad for provenance because they are out-of-band. You run
your experiment in place A and then you copy the results into a spreadsheet in place B.
Even if you are diligent and you include some manual provenance,
e.g., the file path of your input
data in the spreadsheet, there’s no guarantee that someone won’t edit
that file tomorrow. More importantly, nobody can use that spreadsheet alone
to rerun the experiment and get the same results.
With this benchmarking conundrum at hand, I was called in to find a better
alternative. The problem was that I was unsure about what the problem was, or
what a good solution would look like. Thankfully, I wasn’t starting from scratch.
I struggled with benchmarking provenance a lot during my PhD and I never found a
fully satisfactory solution, so it was kind of a nemesis that came back to haunt me.
The ghost of Christmas provenance.
Me, myself, and my provenance
When you do a PhD, you have to do a lot of experiments. If you’re
good and a bit lucky, you get some nice results that you can use
to write a publication. Then you submit this publication and you wait a
few months for peer review. When the reviews come back, you usually need to go
back to your results and use them to answer reviewer questions.
That involves the raw data you used to make the figures in the paper.
In some (unfortunate!) cases, the reviewers might ask you for more experiments.
If your experiments are not reproducible, this
process can quickly turn into a nightmare.
So I had a method. It was not great and it felt like a house of cards but it
served me well for five years without any hiccups, so I’ll tell you all
about it. Or, even better, let’s design it from scratch together.
In the simplest possible terms, an experiment is some code, some binary artifacts, and some
measurements, right?
- What system can keep track of code? Git.
- What system can keep track of measurements? Spreadsheets.
- What system can keep track of binary artifacts? Many. Let’s say tar files.
Great! We have our system. Commit the code. Run the
experiment. Pack everything into a tar archive. Put the measurements, the
path to the tar file and the commit hash of the code into a spreadsheet. If you’re
feeling fancy, you can also record the IDs of the relevant paper figures
in another column. For example:
| Commit |
Date |
Raw Data |
Rate |
Latency |
Figures |
| 14f3d87 |
2026-10-03 |
~/paperA/exp.tar |
30 GB/s |
990 ms |
1, 3 |
BOOM. We have provenance!
So why not use something like that now at ZeroPoint? It obviously worked for me before and
required very little effort to set up. What could go wrong?
Social experiments
Spoiler alert: what could go wrong is other people.
A whole team would now be involved in the benchmarking of the product.
How long do you think it takes before the spreadsheet starts falling apart?
Before people forget to add data, or add some wrong data, or overwrite
other people’s data?
I am a practical person. I considered the option of moving to a remote island
and avoiding other people and the pesky problems they cause. But I persevered.
And I started realizing that I had just stumbled upon the most important
aspect of the problem: control.
More precisely, the lack of control that people should have over the experiments.
People make mistakes. When those mistakes pollute the provenance itself, it
can be impossible to go back. So let’s not allow them to.
- How do you remove control during experiment runs?
Automate everything.
- How do you prevent someone changing the experiment data later?
Make the experiments immutable.
- How do you prevent someone overlooking some dimensions of the experiment?
Make the process systematic.
That sounds promising. Spreadsheets did have one advantage, though.
They were easy to share both with
technical and non-technical people. It would be very nice to keep that property
in the new solution.
To summarize, I wanted automated, immutable, reproducible experiments with
systematic tracking of their dimensions. And easily shareable too. Shouldn’t
be hard to do, right?
Version my data, please
And indeed, it wasn’t. Having defined the problem clearly, the solution
came almost organically. So organically that I don’t really remember
exactly how. With the requirements written down, it would not have been difficult
to create the infrastructure myself. But I
was lucky enough to have someone else do my work for me. In my book, that’s the best kind
of solution!
More specifically, the solution I chose came in the form of a framework called
DVC, which stands for Data
Version Control. Even though it is marketed for AI/ML and data science, it
allowed me to model the benchmarking problem and gave me a very elegant base for
a full provenance solution. DVC defines what an experiment is and models its
provenance in a systematic way. All that remained was to orchestrate
the automated experiment execution and result collection.
In the end, DVC augmented with my own infrastructure made hardware benchmarking a
single-command procedure.
Run your experiment and its data is stored immutably in an artifact store,
while its measurements are pushed to a database and visualized in a dashboard.
Nowhere in the process does a (non-malicious) user have the chance to leave out any
part of the experiment configuration or change the results. What you see is what you get.
I cannot overstate the peace of mind that such infrastructure gives to a team.
Being able to prove which configuration was used to produce a specific measurement
and what post-processing was applied afterward can dramatically increase the confidence
in the results. This, in turn, frees up time for things that matter:
more efficient execution, more experimentation, and more investigation of the
performance of the product itself.
Solving problems by defining problems
Reflecting on this journey, I come back to something that I have been
considering a lot over the years. The more experience I gain, the more it seems that
most of my job is about clearly defining the problem I am trying to solve.
How I arrive at that definition varies, depending on the type of problem.
In this case, it helped to reflect on what I had tried before and why it couldn’t work now.
By the way, such reflection also works great at the team level. Ask “why can’t [the simplest
solution] work in our case?” and then try (really try!) to answer this question honestly.
You might be surprised by the answers you end up with.
When I work on R&D projects, I tend to define the problem by
trying to solve it. Sounds counterintuitive, but the point is not to
succeed. The first attempt usually fails,
but the failure itself usually reveals aspects of the problem
not considered before. I feel that this approach works better in
R&D projects because there are too many “unknown unknowns” for
reflection to give any useful results.
Whatever technique you use, I think that the skill of clearly defining problems is
more important than ever. Nowadays, AI can solve a lot of engineering problems.
With the correct guidance, it can help you solve some really hard ones too.
But in my experience, if you don’t define your problems clearly, AI is not much help.
Without a good problem definition, AI can even make things worse, because it deprives
you of the step-by-step problem-solving process that could have revealed
a wrong problem definition.
When AI inevitably gives you an intricate solution for a different problem than the one you had,
you have wasted time, money, and energy that could have been better spent elsewhere.
So what are you still doing here? Go out there and find some problems to define!
15 Feb 2026
The Nightmare
It’s 3:00 AM. You are woken up by alerts from your data pipeline about a neighborhood-wide blackout.
You look out the window and you see the street bathed in the yellow glow of the street lights.
The neighborhood clearly has power. You are starting to sweat profusely.
Is it a bug in the query? A bug in the firmware? Faulty sensors?
You try to look at the raw data, but the previous inputs have already been dropped
from the limited storage of the smart energy meters.
Your sweat is starting to run cold. What do you do?
The above might not just be a bad dream.
In fact, such “nightmare” scenarios were the motivation for my research
in explainable stream processing systems.
When you are processing millions of datapoints per second, can you go back
and explain why a result is not what you expected?
Is it even possible? And can you do it without destroying the performance of your system?
Let’s find out!
Data Streaming and Explainable Outputs
Data streaming queries extract low-latency insights by processing data items (tuples) in near real-time.
In contrast to relational databases where the data is “stationary” and users run queries on it,
in data streaming there are long-running queries over streams of fast-moving data.
Those queries use transformations called operators (e.g., filter, join, aggregate).
In our nightmare scenario, smart meters in each building or city block could be running
a continuous query that processes per-second voltage and power measurements
and produces an out-of-power event if some conditions are met.
When something goes wrong in a streaming query, it can be critical to know why a specific output was produced.
This question is answered by fine-grained data provenance,
which returns the set of the inputs that contributed to the production of an output event.
Provenance was not a big area in data streaming research in 2018,
when I was six months into my PhD.
At that time, I was taking slow steps on another topic,
custom CPU scheduling for stream processing engines.
My supervisor, Vincenzo Gulisano,
had come across a recent paper on data provenance
and thought we could come up with a better technique.
Looking back, I think he just wanted me to get started on paper writing and performance evaluations.
Maybe he was hoping that we would get a small publication out of it
but without huge expectations for groundbreaking work.
That previous provenance technique
was a simple but functional idea:
- It annotated each tuple with a unique ID.
Every time a new tuple was produced,
it carried the tuple IDs of all of its parents.
- This way, input tuple IDs were propagated downstream to the outputs.
- Provenance was computed with a streaming join between
all the inputs and all the outputs: each input whose ID matched an annotation present
in an output was part of that output’s provenance.
However, we identified several issues:
- The join could be expensive, considering that each output could depend on hundreds or more raw inputs.
- The size of the ID annotations carried by each tuple could grow arbitrarily large (depending on query semantics). In practice, the memory usage of the stream processing engine could explode when recording provenance with this technique.
- All input data had to be kept until it was joined with all possible results. This could be impossible in resource-constrained devices or high-throughput deployments.
The question was, could we do better?
The Lightbulb Moment
Vincenzo’s idea was to avoid the variable-length annotations and the join.
This could be done by constructing a Directed Acyclic Graph (DAG) for provenance.
- The nodes of the provenance graph would be the tuple objects.
The edges would be object references to “parent” (upstream) tuples.
Each output tuple could traverse the graph upstream to find its parents,
eventually discovering its ancestral input tuples.
- A join would no longer be needed. The graph
would continuously maintain the relation between inputs and outputs
throughout the query.
- As a bonus, it was possible to create the graph without variable-length annotations
by taking advantage of the semantics of the query operators.
Each tuple required some small, constant-size metadata.
With the variable-length annotations and the join solved,
we just needed to fix the last problem:
not maintaining all query inputs.
However, when doing aggregations and joins, it is not trivial to decide
when an input has finished contributing to potential outputs somewhere in a streaming query.
To reduce the problem space, we were brainstorming various solutions,
such as traversing the graph for each tuple
to somehow mark tuples that were not part of any output’s provenance.
It was then that I had my “aha moment”.
Having worked in large Java codebases before,
I always had the garbage collector and object references in the back of my mind,
mostly for performance.
As we were discussing reference-based graphs,
something clicked.
Overwhelmed by anticipation, I told Vincenzo (with all the nervousness of a first-year PhD student),
“Wait a minute… What if the system threw away the garbage tuples on its own? I think our problem is already solved!”
Here’s what I really meant:
The query engine we were using (Apache Flink) runs on the Java Virtual Machine (JVM).
On a high level, the JVM garbage collects objects that are not reachable from anywhere in the program.
In our case, input tuples that did not contribute to any output would not be connected in the provenance graph and thus would eventually become unreachable.
Thus, our provenance graph would ensure that only our provenance inputs would be kept alive in the memory of the process. Other irrelevant inputs would be discarded automatically and transparently by the memory management.
Since the provenance graph had no cycles, the same technique would work in languages that use plain reference counting instead of garbage collection, such as C++ (shared_ptr) and Rust (Rc/Arc).
This was great! We could take advantage
of decades of optimizations in garbage collection to clean up our query inputs
and we did not need to dive deeper and analyze how long each input could live for.
When we implemented and evaluated our provenance method,
which we named GeneaLog (hinting at the tuple ancestry),
the results were impressive: we could compute live, eager data streaming provenance with overheads lower than 15% in most cases (where previous techniques would cause very significant slowdowns).
Taking inspiration from the previous streaming provenance work, I implemented the framework to instrument (wrap) existing query operators.
This meant that there was no need for a custom version of the stream processing engine to compute provenance.
It was enough to update your query to use our framework and you were set.
An implementation of GeneaLog is available on GitHub.
The repository contains an extended version of the technique that includes Ananke, my later work on forward streaming provenance.
What Does This All Mean?
As I reflect on that day, I am trying to explain the feeling of excitement
when a great idea just “clicks” into place.
That feeling is often disproportionately stronger than the idea’s objective value.
Maybe it doesn’t come from the idea itself,
but from our ability to synthesize
seemingly unrelated tidbits of knowledge acquired sometime before
and combine them with patterns we recognize in the now
to create something new.
That feeling says that you do not need to plan everything.
You do not need to constantly judge the things you read, watch, learn,
based on some perceived usefulness.
You cannot really predict what could lead you to your next great idea.
You’re allowed to pursue that “pointless” project, to read that “useless” book,
even to sit on the couch and do nothing.
Maybe if you give your mind enough space,
you will find more moments
where great ideas are, indeed, just there.
As for me, that simple insight led to a conference paper (and later a journal extension),
and eventually two more provenance papers at VLDB.
More importantly, it changed how I build systems.
Today, I’ve become a bit of a provenance evangelist in my current work,
constantly asking: “What is the source of truth for this data and how do we keep it?”.
But that is a story for another day.