If I have a collection, I loop it.

foreach (var item in items)
{
    Process(item);
}

It was clean.

Predictable.

Easy to reason about.

And for a long time… it was enough.

Until one day, it wasn’t.

https://www.youtube.com/@DotNetFullstackDev

The First Time “Correct” Code Felt Slow

I remember working on a feature that processed a list of files.

Nothing fancy:

• read file
  

• transform data
  

• save output
  

The code looked fine.

But when the dataset grew, something changed.

It didn’t break.

It just… took forever.

That’s when I first felt the difference between:

code that works

and

code that works under pressure

The Naive Thought: “Let’s Make It Parallel”

The obvious idea came next:

“Why not process multiple items at once?”

That’s when I discovered:

Parallel.ForEach(items, item =>
{
    Process(item);
});

And it felt like magic.

The same logic.

Faster execution.

Problem solved.

At least, that’s what I thought.

When Parallelism Started Breaking Things

Very quickly, subtle issues appeared:

• Random exceptions
  

• Data corruption
  

• Race conditions
  

• Inconsistent results
  

The code didn’t look dangerous.

But it was.

Because I had unknowingly crossed a boundary:

I moved from sequential thinking to concurrent thinking

without changing how I designed my code.

That’s when I realized something uncomfortable:

Parallelism is not just about speed.

It’s about control.

Why foreach Exists (and Why It Still Matters)

Let’s go back for a second.

foreach is not “basic”.

It’s safe by design.

foreach (var item in items)
{
    Process(item);
}

It guarantees:

• one item at a time
  

• predictable order
  

• no shared state conflicts (by default)
  

It’s slow for large workloads, yes.

But it’s calm.

And calm code is easy to trust.

What Parallel.ForEach Really Introduced

Parallel.ForEach didn’t just introduce speed.

It introduced:

• multiple threads
  

• non-deterministic execution order
  

• shared state risks
  

• scheduling complexity
  

Parallel.ForEach(items, item =>
{
    Process(item);
});

Now:

• items may run simultaneously
  

• order is not guaranteed
  

• multiple threads can touch the same data
  

This is where design begins to matter more than syntax.

The First Real Problem: Shared State

Here’s a classic mistake I made:

int total = 0;

Parallel.ForEach(items, item =>
{
    total += item.Value;
});

Looks innocent.

But it’s broken.

Multiple threads updating total at the same time leads to:

• lost updates
  

• incorrect results
  

This is where I first encountered the need for thread-safe constructs.

Enter ConcurrentDictionary — The Controlled Shared State

When multiple threads need to share data, normal collections fail.

That’s where ConcurrentDictionary comes in.

var dict = new ConcurrentDictionary<int, string>();

Parallel.ForEach(items, item =>
{
    dict.TryAdd(item.Id, item.Name);
});

Why this works:

• internal locking or lock-free strategies
  

• atomic operations
  

• safe concurrent access
  

But here’s the deeper realization:

We didn’t just add a new class.

We added discipline to shared data.

But Even That Wasn’t Enough

At some point, I ran into a different problem.

Not just:

• “how do I process items faster?”
  

But:

• “how do I coordinate producers and consumers?”
  

Because now:

• one part of the system generates work
  

• another part processes it
  

And they don’t move at the same speed.

That’s when BlockingCollection entered my world.

BlockingCollection — When Flow Matters More Than Speed

BlockingCollection is not about parallel loops.

It’s about controlled pipelines.

var collection = new BlockingCollection<int>();

// Producer
Task.Run(() =>
{
    foreach (var item in items)
    {
        collection.Add(item);
    }
    collection.CompleteAdding();
});

// Consumer
foreach (var item in collection.GetConsumingEnumerable())
{
    Process(item);
}

What changed here?

• producer and consumer are decoupled
  

• consumer waits when no data is available
  

• producer slows down if buffer is full
  

This is not just concurrency.

This is flow control.

That’s When It Clicked for Me

All these constructs are not random.

They exist because problems evolved.

Let’s map that evolution:

The “Family” (Once It Finally Made Sense)

🟢 foreach

• simple iteration
  

• single-threaded
  

• predictable
  

• safe
  

Used when:

• workload is small
  

• order matters
  

• simplicity is priority
  

🟡 Parallel.ForEach

• multi-threaded processing
  

• CPU utilization
  

• faster execution
  

Used when:

• work is independent
  

• no shared state
  

• CPU-bound tasks
  

🔵 ConcurrentDictionary (and other concurrent collections)

• safe shared state
  

• atomic operations
  

Used when:

• threads must share data
  

• consistency matters
  

Also part of this group:

• ConcurrentQueue
  

• ConcurrentBag
  

• ConcurrentStack
  

🟣 BlockingCollection

• producer-consumer coordination
  

• backpressure handling
  

Used when:

• pipeline systems
  

• ingestion/processing flows
  

• uneven workloads
  

And It Still Didn’t End There…

Once I went deeper, I found more tools in this ecosystem:

🔶 Task.WhenAll (async parallelism)

await Task.WhenAll(items.Select(item => ProcessAsync(item)));

This is:

• asynchronous parallelism
  

• better for IO-bound tasks
  

Not the same as Parallel.ForEach.

🔷 Parallel LINQ (PLINQ)

var results = items
    .AsParallel()
    .Select(x => Process(x))
    .ToList();

A more declarative way of parallelism.

🔶 Channels (System.Threading.Channels)

Modern alternative to BlockingCollection.

More control.

More performance.

🔷 TPL Dataflow (advanced pipelines)

For complex systems:

• multi-stage processing
  

• batching
  

• throttling
  

So… Why So Many?

This question bothered me for a long time:

“Why doesn’t .NET just give one way to do this?”

Because the problem is not one-dimensional.

Each tool solves a different kind of pain

The Real Answer (The One That Stayed With Me)

This is what finally made everything feel calm:

Concurrency is not a feature.

It’s a set of trade-offs.

Every new construct exists because:

• the previous one was not enough
  

• or was too dangerous in certain situations
  

What Changed in My Thinking

I stopped asking:

“Which one is better?”

And started asking:

“What problem am I actually solving?”

Because:

• using Parallel.ForEach for IO work is wrong
  

• using Task.WhenAll for CPU-heavy loops is inefficient
  

• using shared state without concurrent collections is dangerous
  

• using concurrency when not needed is unnecessary complexity
  

Final Thought

At first, this ecosystem feels overwhelming.

Too many options.

Too many APIs.

Too many patterns.

But over time, something shifts.

You stop seeing:

• different classes
  

And start seeing:

• different responsibilities
  

And that’s when everything becomes simpler.

If You Remember One Thing

Start simple.

Add concurrency only when pain demands it.

And when you add it, choose the tool that matches the problem — not the trend.

That’s how this evolution makes sense.

“My job is to write code. Testing is QA’s job. Requirements are BA’s job. Release is DevOps’ job.”

That mindset creates average developers.

High-value developers don’t work like that. They understand one simple reality:

Software is only valuable when it runs correctly in production and solves the right business problem.

To do that, a developer must wear four hats—whether the org admits it or not:

1. Debugger

2. Business Analyst

3. QA (Quality mindset)

4. Release Expert (delivery mindset)

This article explains why, and how you can build these skills without becoming “everything everywhere.”

1) Developer as a Debugger

Because production doesn’t care about your intentions

Anyone can write code that compiles.
A strong developer writes code that survives reality.

Reality includes:

• messy data

• unexpected user behavior

• slow networks

• concurrency issues

• timeouts

• nulls

• partial failures

• race conditions

• bad deployments

• hidden dependencies

When things break, the developer who can debug fast becomes the most valuable person in the room.

What debugging really means (not just breakpoints)

A mature developer uses a layered approach:

A) Reproduce the problem

• “When exactly does it happen?”

• “Is it consistent or random?”

• “Is it environment-specific (prod vs staging)?”

• “What input triggers it?”

B) Reduce the scope

• isolate the smallest set of conditions to trigger it

• remove noise until the problem becomes obvious

C) Observe before changing

• logs

• metrics

• traces

• correlation IDs

• database queries

• upstream/downstream service behavior

D) Hypothesis-driven debugging
Not “try random fixes.”
Instead:

• form a hypothesis

• test it quickly

• confirm/deny with evidence

• move to next hypothesis

Mini example (real-life)

Bug report: “Users say checkout fails sometimes.”

Weak developer:

• adds a try/catch

• retries randomly

• ships and prays

Strong developer:

• checks payment gateway logs

• correlates request IDs

• sees timeouts at peak hours

• identifies missing timeout policy + circuit breaker

• adds proper resilience + observability

• writes a regression test

Debugging is not fixing. Debugging is investigation.

2) Developer as a Business Analyst

Because building the wrong thing perfectly is still failure

Many project disasters are not “technical failures.”

They are misunderstanding failures.

A developer who thinks like a BA asks:

• “Why are we building this?”

• “Who benefits?”

• “What does success look like?”

• “What are the edge cases?”

• “What happens when input is missing?”

• “What should happen when a dependency is down?”

The BA mindset developers need

A) Translate business language into system behavior
Business says: “Approve loans faster.”
System reality:

• what data is required?

• what validations?

• what rules?

• what exceptions?

• what audit logs?

• what SLA?

B) Clarify ambiguity early
If requirements have multiple interpretations, don’t code. Clarify.

C) Confirm with examples
Instead of arguing, ask:

• “Can you give 3 sample scenarios?”

• “What should happen in each?”
  Then you convert it to acceptance criteria.

The biggest BA habit: “Definition of Done”

A good developer doesn’t start coding without:

• expected input

• expected output

• error cases

• success criteria

• non-functional expectations (performance, security)

That’s business analysis inside engineering.

3) Developer as a QA

Because quality is not a department — it’s a mindset

QA is not only about “finding bugs.”

QA is about protecting user trust.

A developer with QA mindset thinks:

• “How can this fail?”

• “How can a user break this?”

• “What assumptions am I making?”

• “What happens if dependency returns garbage?”

• “What happens under load?”

• “What if the clock/timezone changes?”

• “What if the user double-clicks 5 times?”

The QA toolbox a developer must own

You don’t need to become a full-time tester. But you must cover quality basics:

A) Unit tests

• validate logic

• enforce contracts

B) Integration tests

• validate database calls

• validate external service interactions

• validate serialization/deserialization

C) Contract tests (if microservices)

• prevent breaking API changes

D) Regression tests

• every serious bug gets a test so it never returns

E) Exploratory testing mindset
Before shipping:

• try weird inputs

• try large payloads

• try slow network simulation (where possible)

• try “unhappy paths” intentionally

One powerful rule I teach

If you fixed a bug and didn’t add a test, you didn’t really fix it.
You just postponed it.

4) Developer as a Release Expert

Because software doesn’t matter until it’s deployed correctly

Many developers treat release like a ceremony:
“DevOps will handle.”

But production failures often come from:

• wrong configuration

• missing environment variables

• database migration mismatch

• connection strings wrong

• secrets not set

• feature flags misused

• backward-incompatible changes

• no rollback plan

A release-aware developer prevents these.

Release mindset essentials

A) Know your deployment flow
Even if you don’t run it, understand:

• build pipeline stages

• artifact creation

• environment promotions

• approvals

• release notes

B) Config discipline

• Never hardcode environment-specific values

• Use proper configuration providers

• Use secrets management

C) Backward compatibility
If you deploy services independently:

• database changes must be additive first

• APIs must remain compatible

• versioning must be planned

D) Rollback readiness
Every release should answer:

• “If it fails, how do we rollback?”

• “What data changes are irreversible?”

• “Do we have feature flags?”

The difference between a coder and an engineer

Coder: “It works on my machine.”
Engineer: “It will survive production, upgrades, failures, and rollback.”

How These Four Hats Work Together (the real picture)

When you combine these skills, you start shipping like a mature engineer:

• BA mindset prevents building the wrong thing

• QA mindset prevents obvious breakages

• Release mindset prevents deployment disasters

• Debug mindset reduces MTTR when reality still breaks you

This is how senior developers become “go-to” people.

Not because they write clever code.
Because they deliver reliable outcomes.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

A Practical Roadmap (so you actually improve)

Week 1–2: Debugging upgrade

• Learn logs + correlation IDs

• Practice reproducing issues locally

• Document your debugging steps like an investigator

Week 3–4: BA upgrade

• For every task, write 5 clarifying questions before coding

• Convert requirements into “Given / When / Then” scenarios

Week 5–6: QA upgrade

• Add unit tests for new work

• For every bug fix, add a regression test

• Learn integration test basics for your stack

Week 7–8: Release upgrade

• Understand your pipeline end-to-end (Jenkins/Azure DevOps/GitHub Actions)

• Learn feature flags + config management basics

• Learn safe DB migration patterns (expand/contract)

Final Thought

A developer who only codes is easy to replace.

A developer who can:

• debug under pressure,

• clarify business intent,

• protect quality,

• and ship safely,

…becomes a force multiplier.

That’s not “extra work.”

That’s what professional software engineering actually is.

No.

In real projects, your code isn’t “done” until Jenkins proves it can build, test, and ship it.
And the Jenkinsfile is the contract that defines that proof.

If your Jenkinsfile is weak, your delivery is weak. Simple.

https://www.youtube.com/@DotNetFullstackDev

What is a Jenkinsfile (in plain words)

A Jenkinsfile is a text file (checked into your repo) that tells Jenkins:

• how to pull the code

• how to restore dependencies

• how to build

• how to run tests

• how to publish artifacts

• how to deploy (optional)

It turns your build pipeline into version-controlled code.

So instead of “Jenkins UI configuration jungle”, you get Pipeline as Code.

Where it sits in a .NET repo

Typical structure:

MySolution/
  src/
    MyApi/
    MyLib/
  tests/
    MyApi.Tests/
  Jenkinsfile
  MySolution.sln

Put Jenkinsfile at repo root unless you have a strong reason not to.

What a good Jenkins pipeline does for .NET

A clean pipeline usually has these stages:

1. Checkout

2. Restore

3. Build

4. Test (with coverage)

5. Publish (build output)

6. Archive artifacts

7. Optional: Quality gates (Sonar), security scans, docker build, deploy

This is not “extra”.

This is basic engineering hygiene.

A solid Jenkinsfile for .NET (Linux agent)

This is a working baseline for most teams.

pipeline {
  agent any

  options {
    timestamps()
    ansiColor(’xterm’)
    skipDefaultCheckout(true)
    buildDiscarder(logRotator(numToKeepStr: ‘30’))
  }

  environment {
    DOTNET_CLI_TELEMETRY_OPTOUT = ‘1’
    DOTNET_NOLOGO = ‘1’
    CONFIGURATION = ‘Release’
    SOLUTION = ‘MySolution.sln’
    TEST_RESULTS_DIR = ‘TestResults’
  }

  stages {

    stage(’Checkout’) {
      steps {
        checkout scm
        sh ‘dotnet --info’
      }
    }

    stage(’Restore’) {
      steps {
        sh “dotnet restore ${SOLUTION}”
      }
1    }

    stage(’Build’) {
      steps {
        sh “dotnet build ${SOLUTION} -c ${CONFIGURATION} --no-restore”
      }
    }

    stage(’Test’) {
      steps {
        sh “”“
          mkdir -p ${TEST_RESULTS_DIR}
          dotnet test ${SOLUTION} -c ${CONFIGURATION} --no-build \
            --logger “trx;LogFileName=test-results.trx” \
            --results-directory ${TEST_RESULTS_DIR}
        “”“
      }
      post {
        always {
          junit allowEmptyResults: true, testResults: “${TEST_RESULTS_DIR}/**/*.trx”
          archiveArtifacts allowEmptyArchive: true, artifacts: “${TEST_RESULTS_DIR}/**/*”
        }
      }
    }

    stage(’Publish’) {
      steps {
        sh “”“
          rm -rf out || true
          dotnet publish src/MyApi/MyApi.csproj -c ${CONFIGURATION} --no-build -o out/MyApi
        “”“
      }
    }

    stage(’Archive Build Output’) {
      steps {
        archiveArtifacts artifacts: “out/**”, fingerprint: true
      }
    }
  }

  post {
    success {
      echo “✅ Build + Test + Publish succeeded.”
    }
    failure {
      echo “❌ Pipeline failed. Fix it before you touch anything else.”
    }
    always {
      cleanWs(deleteDirs: true, disableDeferredWipeout: true)
    }
  }
}

What this does (in fresher language)

• restore: downloads NuGet packages

• build: compiles solution

• test: runs tests + stores results

• publish: creates deployable output (dlls + config)

• archive: stores the output in Jenkins as a downloadable artifact

Windows agent version (if your org uses Windows nodes)

Just replace sh with bat and path formatting.

pipeline {
  agent any

  options {
    timestamps()
    skipDefaultCheckout(true)
    buildDiscarder(logRotator(numToKeepStr: ‘30’))
  }

  environment {
    DOTNET_CLI_TELEMETRY_OPTOUT = ‘1’
    DOTNET_NOLOGO = ‘1’
    CONFIGURATION = ‘Release’
    SOLUTION = ‘MySolution.sln’
    TEST_RESULTS_DIR = ‘TestResults’
  }

  stages {

    stage(’Checkout’) {
      steps {
        checkout scm
        bat ‘dotnet --info’
      }
    }

    stage(’Restore’) {
      steps {
        bat “dotnet restore %SOLUTION%”
      }
    }

    stage(’Build’) {
      steps {
        bat “dotnet build %SOLUTION% -c %CONFIGURATION% --no-restore”
      }
    }

    stage(’Test’) {
      steps {
        bat “”“
          if not exist %TEST_RESULTS_DIR% mkdir %TEST_RESULTS_DIR%
          dotnet test %SOLUTION% -c %CONFIGURATION% --no-build ^
            --logger “trx;LogFileName=test-results.trx” ^
            --results-directory %TEST_RESULTS_DIR%
        “”“
      }
      post {
        always {
          junit allowEmptyResults: true, testResults: “%TEST_RESULTS_DIR%/**/*.trx”
          archiveArtifacts allowEmptyArchive: true, artifacts: “%TEST_RESULTS_DIR%/**/*”
        }
      }
    }

    stage(’Publish’) {
      steps {
        bat “”“
          if exist out rmdir /s /q out
          dotnet publish src\\MyApi\\MyApi.csproj -c %CONFIGURATION% --no-build -o out\\MyApi
        “”“
      }
    }

    stage(’Archive Build Output’) {
      steps {
        archiveArtifacts artifacts: “out/**”, fingerprint: true
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

The “real-world” upgrades senior teams add

1) Branch-based behavior (main vs feature)

You don’t deploy feature branches.

when { branch ‘main’ }

Example:

stage(’Deploy’) {
  when { branch ‘main’ }
  steps {
    echo “Deploying only from main...”
  }
}

2) Multi-branch Pipeline (how teams run Jenkins in 2026)

In Jenkins, you create a Multibranch Pipeline job.

Then Jenkins automatically:

• finds Jenkinsfile in each branch

• runs PR builds

• shows status on PR

This is how professional teams work. Not “click-build”.

3) Code coverage (real expectation)

Add coverage collector (example using coverlet built-in support):

dotnet test MySolution.sln -c Release --no-build \
  /p:CollectCoverage=true \
  /p:CoverletOutputFormat=cobertura \
  /p:CoverletOutput=TestResults/coverage/

Then publish cobertura report via a plugin if your Jenkins has it.

4) SonarQube Quality Gate (common in enterprises)

Typical flow:

• Begin analysis

• Build + Test

• End analysis

• Wait for quality gate

(Your exact code depends on how Sonar is configured in your org, but the concept is always this.)

5) Docker build (modern microservices)

If you ship containers:

stage(’Docker Build’) {
  steps {
    sh ‘docker build -t myapi:${BUILD_NUMBER} -f src/MyApi/Dockerfile .’
  }
}

Common fresher mistakes (and why seniors get angry)

❌ Mistake 1: Restore every time without caching

Your builds get slow, people stop trusting CI, and then quality dies.

Fix: Use Jenkins workspace caching or a persistent agent home so NuGet cache survives across builds.

❌ Mistake 2: “Build passed so we can skip tests”

Trash mindset.

Tests are part of the build.

❌ Mistake 3: Publishing from random branches

That’s how you deploy garbage to production.

❌ Mistake 4: Secrets inside Jenkins file

Never hardcode:

• connection strings

• API keys

• tokens

Use Jenkins Credentials + environment injection.

What I tell every fresher on Day 1

If you want to become a strong .NET engineer:

• Learn your code ✅

• Learn your build pipeline ✅

• Learn how the artifact is produced ✅

• Learn how it gets deployed ✅

Because at senior level, you’re not paid for “writing code”.

You’re paid for delivering reliable software.

There’s a quiet fear moving through development teams.

It’s not loud.
It’s not dramatic.
But it’s there.

You can feel it in conversations.

“AI can generate clean architecture.”
“AI knows all design patterns.”
“AI can design scalable systems.”

And then comes the real question:

If AI is preloaded with patterns and architectures…
Why should I memorize Singleton, Factory, CQRS, DDD, Clean Architecture, or CAP theorem?

Let’s answer this honestly.

Not defensively.

Not emotionally.

But structurally.

AI Knows Patterns. It Does Not Understand Trade-offs.

This is the first distinction developers must internalize.

AI has seen:

• Millions of examples of Repository pattern

• Thousands of Clean Architecture implementations

• Countless microservice templates

• Infinite Singleton mistakes

It can reproduce patterns.

But reproduction is not judgment.

When you design a system, you are not selecting patterns because they exist.

You are selecting patterns because:

• You understand constraints.

• You understand failure modes.

• You understand scale.

• You understand domain complexity.

• You understand operational cost.

AI suggests patterns based on statistical frequency.

Architects choose patterns based on contextual necessity.

That difference is everything.

Memorizing Patterns Is Not About Recall. It’s About Mental Models.

Let’s be honest.

No senior engineer memorizes patterns like flashcards.

They internalize mental models.

When a senior hears:

“High read, low write, multi-region system with eventual consistency.”

Their brain automatically activates:

• Caching strategy

• Read replicas

• CQRS possibility

• Event-driven architecture

• CAP trade-offs

• Idempotency design

That reflex does not come from memorization.

It comes from structured understanding.

AI cannot replace your mental model.

It can only generate based on its training.

Without internal mental models, you cannot evaluate AI’s output.

AI Is a Pattern Reproducer. Developers Are Context Interpreters.

Let’s compare.

AI:

• Detects pattern similarity

• Generates common architecture

• Suggests typical solutions

Developer:

• Evaluates non-functional requirements

• Considers team skill level

• Accounts for deployment constraints

• Predicts operational burden

• Assesses long-term maintainability

• Evaluates organizational culture

Architecture is not just technical.

It is organizational.

AI cannot read your org chart.

Clean Architecture Is Not Code Structure. It’s Responsibility Separation.

AI can generate a Clean Architecture folder structure.

It cannot ensure:

• Proper dependency direction

• Business logic isolation

• Domain purity

• Real boundary enforcement

• Team discipline over time

Clean Architecture is less about code and more about thinking.

Without understanding dependency inversion, AI-generated layers become cosmetic.

And cosmetic architecture collapses under complexity.

System Design Is Decision-Making Under Uncertainty

System design exists because of trade-offs.

Latency vs consistency.
Cost vs scalability.
Simplicity vs flexibility.
Performance vs readability.

AI can suggest a “scalable architecture.”

But scalable for:

• 1,000 users?

• 1 million users?

• Global?

• Real-time?

• Batch?

• Regulated industry?

AI does not own consequences.

Architects do.

That is why system design cannot disappear.

The Illusion of “Preloaded Knowledge”

Yes, AI has been trained on:

• Martin Fowler

• Uncle Bob

• Microsoft docs

• StackOverflow

• GitHub repositories

But knowledge is not ownership.

AI doesn’t:

• Refactor your production monolith safely

• Debug your memory leak in staging

• Sit through incident postmortems

• Feel SLA pressure

• Answer to executives when systems fail

Architecture knowledge matters because consequences are real.

Why Developers Still Need to Learn Design Patterns

Because patterns are not templates.

They are:

• Reusable solutions to recurring problems

• Vocabulary for technical communication

• Shared understanding across teams

• Conceptual compression tools

When you say:

“Let’s apply Strategy pattern here.”

You compress an entire explanation into two words.

If you rely only on AI and never internalize patterns:

You cannot communicate at that abstraction level.

And communication is seniority.

AI Amplifies Good Architects. It Exposes Weak Ones.

Here is the uncomfortable truth.

AI will not replace strong architects.

It will expose shallow ones.

If you don’t understand:

• Thread safety

• Transaction boundaries

• Data consistency

• Domain modeling

• SOLID principles

You cannot detect when AI gives you a flawed design.

You will accept incorrect patterns confidently.

And that is more dangerous than ignorance.

The Real Shift: From Pattern Recall to Pattern Evaluation

In the AI era, developers do not need to memorize every pattern detail.

But they must:

• Recognize patterns

• Understand trade-offs

• Evaluate applicability

• Modify for context

• Reject inappropriate suggestions

AI proposes.

Architects dispose.

The Role of AI in Modern Architecture

AI becomes:

• Idea generator

• Boilerplate accelerator

• Edge-case enumerator

• Documentation summarizer

• Alternative approach suggester

But AI is not:

• Risk owner

• Accountability bearer

• Context interpreter

• Long-term maintainer

Architecture survives because responsibility survives.

What Happens If You Stop Learning Patterns?

You become dependent.

Dependent engineers:

• Cannot evaluate suggestions

• Cannot design from first principles

• Cannot debug architectural failures

• Cannot scale systems independently

• Cannot lead technical decisions

You become an operator, not an engineer.

And operators are automatable.

What Actually Changes in the AI Era

Earlier:
You memorized patterns to recall them manually.

Now:
You learn patterns to validate AI output.

The reason changes.
The necessity does not.

Architecture Is Thinking, Not Syntax

Clean architecture is not about folder structure.

It is about:

• Separation of concerns

• Dependency flow

• Domain independence

• Boundary protection

• Future-proofing

AI can scaffold layers.

Only engineers protect boundaries.

Final Perspective

Design patterns are not dying.

They are becoming more important.

Because in a world where AI generates code instantly:

The ability to judge architecture becomes the differentiator.

The question is not:

Does AI know design patterns?

The question is:

Do you know enough to recognize when AI misapplies them?

And that, my friend, is what keeps developers relevant.

Sit down.

Close StackOverflow for five minutes.

You asked me yesterday:

“Sir… how did you move from normal backend development into AI systems?
Did you learn machine learning?”

I smiled because every junior asks the wrong question first.

No.

I didn’t learn AI.

I learned where AI fits inside software.

Let me tell you how it actually happened

.

Chapter 1 — The Day I Realized Coding Was Not the Job

When I started my career, I believed software engineering meant writing code.

Controllers.
Repositories.
Stored procedures.
Fixing null reference exceptions at 2 AM.

I optimized queries.
Reduced API latency.
Fought deadlocks.

And I thought:

“This is engineering.”

Then something strange happened.

Every system I built started looking identical.

Different business.
Same architecture.

Controller
 → Service
   → Repository
     → Database

Only nouns changed.

Customer.

Invoice.

Order.

Ticket.

Same patterns.

That’s when I understood something dangerous:

Coding is repetition wrapped in logic.

And repetition is always the first thing automation replaces.

Chapter 2 — The Arrival of the “Alien Words”

Then came the meetings.

Someone said:

• LLM

• GPT

• Agents

• Embeddings

• AI workflows

Half the room nodded confidently.

The other half Googled secretly.

I was in the second half.

It sounded like the industry suddenly switched from C# to astrophysics.

My first instinct?

Ignore it.

Exactly like senior developers ignored:

• async/await

• cloud computing

• containers

History repeats.

Chapter 3 — My First Mistake

I tried learning AI the wrong way.

I opened YouTube.

Suddenly I was watching:

• neural networks

• gradient descent

• backpropagation

• tensor mathematics

After two days I closed everything.

Because I realized:

This is not my layer.

A database engineer doesn’t build SSD firmware.

A web developer doesn’t design TCP/IP.

Why was I trying to become an AI researcher?

Wrong abstraction level.

Chapter 4 — The Moment Everything Clicked

One evening I asked a different question.

Not:

“How does AI work?”

But:

“How would I integrate this into an ASP.NET application?”

That changed everything.

Because suddenly AI looked familiar.

It wasn’t intelligence.

It was an API.

Just another dependency.

Like SQL Server once was.

Chapter 5 — My First Real Experiment

I wrote a tiny endpoint.

POST /summarize

Input:

A long document.

Backend:

Call LLM API.

Return:

Summary.

That’s it.

No magic.

But something felt different.

For the first time…

My API wasn’t executing logic.

It was reasoning.

And I realized:

The future backend doesn’t just process data.

It interprets intent.

Chapter 6 — Understanding the True Role of a Developer

Listen carefully.

AI doesn’t remove engineering.

It moves responsibility upward.

Earlier, I decided:

• how data flows

• how logic executes

Now I decide:

• what AI is allowed to do

• what AI must never do

• how outputs are validated

• how decisions are audited

The LLM became junior.

I became architect again.

Chapter 7 — The Agent Revelation

You know what an AI Agent really is?

Not Jarvis.

Not consciousness.

It’s this:

while(goalNotAchieved)
{
    DecideNextStep();
    ExecuteTool();
    ObserveResult();
}

That’s it.

A loop.

We’ve written this our entire careers.

Background workers.

Schedulers.

Workflow engines.

Only difference?

Decision-making moved from if-statements to probability.

Chapter 8 — The Hard Lesson

My first agent failed spectacularly.

It called tools endlessly.

Generated nonsense.

Burned API credits.

I learned something critical:

AI without guardrails behaves like an infinite while loop.

So I added:

• step limits

• schema validation

• tool allowlists

• timeout policies

• audit logging

Suddenly the system stabilized.

Not because AI improved.

Because engineering improved.

👉I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

Chapter 9 — The Biggest Mindset Upgrade

Junior developers think AI writes software.

Experienced developers realize:

AI increases the importance of architecture.

Because now systems must handle uncertainty.

You don’t trust outputs blindly.

You design verification layers.

Like input validation evolved into output validation.

Chapter 10 — When I Became AI-Augmented

The transition wasn’t dramatic.

No certification.

No career switch.

One day I noticed:

Instead of asking,

“How do I code this feature?”

I asked,

“Should this be deterministic logic or probabilistic reasoning?”

That question changed my design decisions forever.

Some problems belong to code.

Some belong to AI.

Knowing the difference is the new seniority.

Chapter 11 — What I Told My Team

I gathered juniors and said:

Stop fearing AI.

You already know 80% required skills:

• APIs

• scalability

• observability

• security

• async workflows

• distributed systems

AI engineers without backend knowledge struggle.

Backend engineers who learn AI orchestration thrive.

Chapter 12 — The Truth Nobody Says

AI will not replace developers.

But developers who only translate requirements into CRUD APIs?

Yes.

Because that layer is automatable.

The future developer designs systems where:

• humans define goals

• AI proposes solutions

• software enforces correctness

Chapter 13 — My New Architecture Diagram

Earlier:

User → API → Business Logic → DB

Now:

User
 → API
 → AI Planner
 → Tool Layer
 → Validation
 → Database
 → Audit

Same foundation.

Higher abstraction.

Chapter 14 — What I Actually Learned

Not AI.

I learned:

• uncertainty engineering

• intelligent orchestration

• system supervision

• probabilistic integration

AI didn’t replace my skills.

It amplified them.

Chapter 15 — What I Want You To Understand

You don’t transition by abandoning .NET.

You transition by asking:

Where should intelligence live in my system?

Once you see that…

AI stops being alien.

It becomes another service.

Another dependency injection.

Another architectural decision.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final Words (From Senior to Intern)

You asked how I transformed myself.

I didn’t chase AI.

I expanded my definition of software.

The compiler still runs.

The API still scales.

The database still persists.

But now…

My systems can reason.

And I remain the engineer controlling them.

One of the most common and least understood causes of correlated stress is synchronized timing.

Jitter exists to break that synchronization.

This article explains:

• What jitter actually is (beyond the buzzword)

• Why exponential backoff without jitter is dangerous

• Where jitter should be applied

• The math behind load smoothing

• How to implement it correctly in .NET

• When not to use it

This is a focused technical deep dive.

1. What Is Jitter?

In distributed systems, jitter is:

Controlled randomness applied to timing decisions to prevent synchronized behavior.

It is typically added to:

• Retry delays

• Backoff strategies

• Cache expiration times

• Token refresh intervals

• Distributed lock retries

• Heartbeat intervals

Jitter is not about randomness for its own sake. It is about reducing correlated load spikes.

2. The Real Problem: Correlated Retries

Consider a service that depends on an external API.

When the dependency fails temporarily, your application retries using exponential backoff:

Attempt 1 → 2 seconds  
Attempt 2 → 4 seconds  
Attempt 3 → 8 seconds  
Attempt 4 → 16 seconds

This looks correct.

However, if 5,000 clients experience failure at roughly the same time, they will retry in synchronized waves:

• All retry at 2 seconds

• All retry at 4 seconds

• All retry at 8 seconds

This creates retry storms.

Retry storms amplify load during recovery windows.

Instead of allowing the dependency to recover, the system repeatedly overloads it.

This is not a retry problem.

It is a synchronization problem.

3. Why Exponential Backoff Alone Is Insufficient

Exponential backoff increases delay between attempts.
But it does not desynchronize clients.

If all clients calculate:

TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));

They will retry at identical timestamps.

The system load graph will look like:

|      |
|      |
|  /\  |
| /  \ |
|/    \|

Spikes. Silence. Bigger spikes.

Spikes cause cascading failures.

4. Jitter Breaks Correlation

👉 I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

Instead of retrying at a fixed delay, introduce randomness within the backoff window.

There are several jitter strategies.

4.1 Simple Additive Jitter

Add small randomness to fixed delay.

var baseDelay = TimeSpan.FromSeconds(4);
var jitter = TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000));

await Task.Delay(baseDelay + jitter);

This reduces perfect alignment but still clusters around the base delay.

4.2 Full Jitter (Recommended)

Instead of:

delay = exponential

Use:

delay = random(0, exponential)

Implementation:

var retryAttempt = 3;
var exponentialMs = Math.Pow(2, retryAttempt) * 1000;

var delay = Random.Shared.Next(0, (int)exponentialMs);

await Task.Delay(delay);

This spreads retries across the entire backoff window.

It significantly reduces peak concurrency.

This pattern is widely used in large-scale distributed systems.

5. Using Jitter With Polly in .NET

Polly is commonly used for retry policies.

Install:

dotnet add package Polly

5.1 Retry Without Jitter (Not Recommended at Scale)

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(5, retryAttempt =>
        TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

This will cause synchronized retry waves.

5.2 Retry With Full Jitter (Recommended)

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(5, retryAttempt =>
    {
        var maxDelay = Math.Pow(2, retryAttempt);
        var jitterFactor = Random.Shared.NextDouble();

        return TimeSpan.FromSeconds(maxDelay * jitterFactor);
    });

This ensures retries are distributed across the window.

6. Jitter in Caching Strategies

Synchronized cache expiry can cause database overload.

Example:

• Cache TTL = 30 minutes

• All instances populate cache at startup

• All expire at exactly 30 minutes

Result:

• Sudden database surge

• CPU spike

• Latency increase

Correct approach:

var baseTtl = TimeSpan.FromMinutes(30);
var jitterMinutes = Random.Shared.Next(0, 5);

var finalTtl = baseTtl + TimeSpan.FromMinutes(jitterMinutes);

Now expiration spreads over a time range.

Load becomes smoother.

7. Jitter in Distributed Locks

Consider multiple nodes retrying to acquire a distributed lock (e.g., Redis lock).

Without jitter:

• All retry at fixed interval.

• Collisions repeat.

• Lock starvation increases.

With jitter:

var baseDelay = TimeSpan.FromMilliseconds(200);
var jitter = TimeSpan.FromMilliseconds(Random.Shared.Next(0, 150));

await Task.Delay(baseDelay + jitter);

Lock acquisition attempts desynchronize.

Throughput improves.

8. Mathematical Perspective

Assume 10,000 clients retry after 4 seconds.

Without jitter:

• Peak load at 4-second mark ≈ 10,000 requests simultaneously.

With jitter across 4-second window:

• Average ≈ 2,500 per second.

• Peak significantly reduced.

Same total work.
Lower instantaneous stress.

Distributed systems fail due to peak pressure, not average load.

9. Where Jitter Should Be Considered Mandatory

In high-scale backend systems, jitter should be evaluated for:

• HTTP retry policies

• Message queue retries

• Circuit breaker half-open attempts

• Cache expiration

• Token refresh

• Scheduled jobs

• Distributed coordination loops

If timing exists and multiple nodes share similar schedules, jitter likely improves stability.

10. When Not to Use Jitter

Avoid jitter in:

• Hard real-time systems

• Deterministic control systems

• Time-sensitive trading algorithms

• Strict ordering protocols

In these cases, randomness may violate correctness guarantees.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

11. Engineering Maturity Model

Level 1:
Retry with fixed delay.

Level 2:
Retry with exponential backoff.

Level 3:
Retry with exponential backoff + jitter.

Level 4:
Analyze synchronization patterns across the entire system.

Jitter is not a feature.

It is part of distributed system hygiene.

Final Takeaway

If your system:

• Retries failures

• Caches aggressively

• Scales horizontally

• Coordinates distributed nodes

Then synchronized timing is a hidden risk.

Jitter reduces correlated load.
It prevents retry storms.
It smooths peak pressure.
It stabilizes recovery behavior.

In distributed architecture, small randomness can produce significant reliability gains.

I used to think these three were basically the same topic:

• Windows Service

• Kestrel

• HTTP.sys

I thought:

“They’re all just ways to host my .NET API, right?”

That misunderstanding cost me a full day in debugging.

Because they are not the same layer.

They solve different problems.

And until you understand the layering, you’ll keep building apps that run on your laptop… but fail on the server.

This is my real story of how I finally understood it — and how you can implement each one properly.

The real requirement that started it all

My project had a very normal enterprise requirement:

1. Every night, files are dropped into a folder (CSV/Excel)

2. A service should pick them up, validate them, and push them into SQL

3. Operations team wants a simple internal API endpoint:

   • “What files were processed today?”

   • “What failed?”

   • “How many rows were inserted?”

4. The server is Windows.

5. Nobody wants IIS complexity unless needed.

6. Security team later asks: “We want Windows authentication.”

So I ended up using all three:

• Windows Service (for the worker)

• Kestrel (for a simple API)

• HTTP.sys (when Windows-auth + enterprise constraints came in)

Part 1 — Windows Service (Worker)

Why I needed it (and why console apps are a trap)

My first build was a console app.

It ran perfectly.

But the moment I:

• closed the window

• logged out

• restarted the server

…the processing stopped.

That’s when I learned the key difference:

A console app is “someone running it”.

A Windows Service is “Windows running it”.

A Windows Service is designed for:

• automatic start at boot

• running without a logged-in user

• clean stop/restart control

• monitoring and recovery rules

Basically: production reliability.

Implementation — Step-by-step with real explanations

Step 1 — Create a Worker Service project

dotnet new worker -n FileProcessorService
cd FileProcessorService

Why this template?
Because this template is not “just code that runs.” It gives you:

• a hosted lifecycle (start/stop)

• built-in dependency injection

• structured logging

• a background processing model designed for servers

This is the correct base for anything that must run 24/7.

Step 2 — Add Windows Service integration

dotnet add package Microsoft.Extensions.Hosting.WindowsServices

What does this package actually do?
Without this package, your worker is just a normal .NET process.

With this package:

• your app understands Windows Service Control Manager (SCM)

• it receives Start, Stop, Shutdown signals properly

• it behaves like a real service (not like a console app pretending to be one)

This is the difference between:

• “I can run it”
  and

• “IT can operate it”

Step 3 — Configure Program.cs (and why each line matters)

using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

IHost host = Host.CreateDefaultBuilder(args)
    .UseWindowsService(options =>
    {
        options.ServiceName = "FileProcessorService";
    })
    .ConfigureLogging(logging =>
    {
        logging.ClearProviders();
        logging.AddConsole();   // useful during local debugging
        logging.AddEventLog();  // useful when running as service
    })
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();
    })
    .Build();

await host.RunAsync();

Let me explain what’s happening here like I wish someone explained to me:

• CreateDefaultBuilder sets up the standard hosting environment:

  • configuration (appsettings.json + environment variables)

  • dependency injection container

  • logging pipeline

• UseWindowsService tells the host:

  • “Don’t run like a console app”

  • “Run as a Windows service with SCM integration”

• Logging:

  • AddConsole() helps you when you run locally (dotnet run)

  • AddEventLog() is critical because once it runs as a service, you won’t see console output.
    Without EventLog, you’ll feel blind in production.

Step 4 — Write Worker.cs correctly (this is where most people mess up)

using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

public sealed class Worker : BackgroundService
{
    private readonly ILogger<Worker> _logger;

    public Worker(ILogger<Worker> logger) => _logger = logger;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Service started at {time}", DateTimeOffset.Now);

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                _logger.LogInformation("Checking folder for new files...");
                ProcessFiles();
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "File processing failed");
            }

            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }

        _logger.LogInformation("Service stopping at {time}", DateTimeOffset.Now);
    }

    private void ProcessFiles()
    {
        // Real implementation:
        // - scan folder
        // - lock file while processing
        // - validate and parse rows
        // - insert into SQL
        // - move file to Processed/Failed folders
    }
}

Why the CancellationToken matters so much?
Because when Windows stops a service, it doesn’t “kill it immediately.”
It sends a stop request.

If your code ignores cancellation:

• your service takes too long to stop

• it may get force-killed

• it may corrupt processing state

This is how production issues happen.

Step 5 — Publish for deployment

dotnet publish -c Release -r win-x64 --self-contained false -o C:\Services\FileProcessorService

Why publish?
Because servers should not build your project.
Servers should run compiled artifacts.

Publishing creates the final executable + dependencies in a clean folder.

Step 6 — Install as Windows Service

PowerShell (Admin):

New-Service `
  -Name "FileProcessorService" `
  -BinaryPathName "C:\Services\FileProcessorService\FileProcessorService.exe" `
  -DisplayName "File Processor Service" `
  -StartupType Automatic

Now Windows owns it.

Your app becomes an operating system-managed process.

Step 7 — Start and validate

Start-Service FileProcessorService
Get-Service FileProcessorService

Then check logs:

• Windows Event Viewer → Windows Logs → Application

This step matters because:

• most failures happen at startup due to config/path permissions

• EventLog is where you’ll find the truth

Part 2 — Kestrel (Web API)

Why I used it first for the internal status API

Once the worker was stable, Ops team asked:

“Can we have an endpoint to see status?”

I didn’t need enterprise auth initially.
I just needed:

• /health

• /processed-files/today

• /failed-files/today

For this, Kestrel is perfect.

What Kestrel really is

Kestrel is the default web server inside ASP.NET Core.

It is:

• fast

• modern

• easy

• cross-platform

It listens directly on a port like

http://localhost:5070

.

Implementation — Step-by-step with explanation

Step 1 — Create Web API

dotnet new webapi -n FileStatusApi
cd FileStatusApi

Step 2 — Make it run as Windows Service

dotnet add package Microsoft.Extensions.Hosting.WindowsServices

Step 3 — Configure Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Host.UseWindowsService(options =>
{
    options.ServiceName = "FileStatusApiService";
});

builder.Services.AddControllers();

var app = builder.Build();

app.MapGet("/health", () => Results.Ok("OK"));
app.MapControllers();

app.Run();

Key understanding:
Windows Service here controls lifetime.
Kestrel controls HTTP serving.

Different jobs. Same process.

Step 4 — Set explicit port (don’t gamble)

appsettings.json:

{
  "Kestrel": {
    "Endpoints": {
      "Http": {
        "Url": "http://localhost:5070"
      }
    }
  }
}

Why explicit?
In production you need predictability:

• firewall rules depend on known ports

• monitoring depends on known endpoints

• avoiding port collisions requires control

Step 5 — Publish & install like earlier

dotnet publish -c Release -r win-x64 -o C:\Services\FileStatusApi

New-Service `
  -Name "FileStatusApiService" `
  -BinaryPathName "C:\Services\FileStatusApi\FileStatusApi.exe" `
  -StartupType Automatic

Test:

• http://localhost:5070/health

At this point, I thought: “Done.”

Then security came in.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Part 3 — HTTP.sys

Why Kestrel wasn’t preferred in my enterprise environment

Security requirement changed everything:

“Use Windows Authentication. Domain users only.”

Now, yes — you can do Windows Auth with Kestrel, but in many enterprises:

• infrastructure teams prefer OS-level HTTP handling

• they want URL reservations

• they want certificate bindings controlled centrally

• they want kernel-mode handling

That’s where HTTP.sys comes in.

What HTTP.sys really is

HTTP.sys is Windows’ own HTTP stack.

It’s not “a library web server.”
It is an OS feature.

That means:

• Windows controls the port

• Windows controls auth

• Windows controls TLS binding

Your app hooks into it.

Implementation — Step-by-step with explanation

Step 1 — Enable HTTP.sys hosting in Program.cs

using Microsoft.AspNetCore.Server.HttpSys;

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.UseHttpSys(options =>
{
    options.UrlPrefixes.Add("http://localhost:5090");

    options.Authentication.Schemes =
        AuthenticationSchemes.Negotiate |
        AuthenticationSchemes.NTLM;

    options.Authentication.AllowAnonymous = false;
});

var app = builder.Build();

app.MapGet("/secure", (HttpContext ctx) =>
{
    return $"Hello {ctx.User.Identity?.Name}";
});

app.Run();

What this does:

• Requests authentication at the OS HTTP layer

• Your API automatically sees the authenticated user

• No tokens, no login screen, no custom auth middleware required

Step 2 — The critical step: URL reservation (URLACL)

netsh http add urlacl url=http://localhost:5090/ user=Everyone

Why is this needed?
HTTP.sys protects URL prefixes.
If your service runs under a normal account, Windows may block it from binding that URL unless reserved.

Without this:

• service might fail on start

• you’ll waste time blaming your code

• but the problem is OS permission

Step 3 — Publish & install as service

Same as before.

Test:

• http://localhost:5090/secure

• It should return your domain identity automatically.

👉 I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

Final clarity — the mental model I now use

If you remember only this, you’ll never get confused again:

Windows Service = “How my app runs”

Kestrel = “How my app serves HTTP (cross-platform)”

HTTP.sys = “How Windows serves HTTP (OS-level)”

Different layers.

Not competitors.

For a long time, these words floated around my career like background noise.

HTTP
HTTPS
SSL
TLS

I used them.
Configured them.
Enabled HTTPS because “security team said so.”

But if someone had asked me early on:

“Why do we actually need all of this?”
“What really happens when you type a URL and hit Enter?”

I would’ve given a half-confident answer and quietly hoped the conversation moved on.

Not because I was careless —
but because everything worked, and that was enough.

Until one day, it wasn’t.

The Moment I Realized I Was Guessing

The real realization didn’t come from a book.

It came from a production issue.

A website was loading… but behaving strangely.
APIs were reachable, but cookies weren’t sticking.
Some users saw warnings. Some didn’t.

Someone asked:

“Is this HTTP or HTTPS?”
“Which TLS version is negotiated?”
“Is the certificate chain correct?”

I nodded.

Then I opened the browser dev tools and realized something uncomfortable:

I had been using the web, not understanding it.

That’s when I decided to slow down and actually follow what happens when a browser hits a website.

It All Starts With a Lie We Tell Ourselves

We often say:

“The browser calls the server.”

When you type:

https://example.com

You’re not “calling a server”.

You’re triggering a sequence of negotiations, lookups, handshakes, and agreements — most of which happen before any application code runs.

And that’s why HTTP alone was never enough.

Step 1: The Browser Doesn’t Know Where the Website Lives

The first truth:

The browser has no idea where example.com is.

So the very first thing it does is ask:

“Who knows the IP address for this name?”

That’s a DNS lookup.

No HTTP yet.
No TLS yet.
Just name resolution.

Only after DNS answers does the browser know which machine to talk to.

Step 2: The Browser Opens a Raw Connection

Now the browser has an IP address.

So it opens a TCP connection.

This is still not HTTP.
This is still not HTTPS.

It’s just:

“Hey, I can reach you.”

At this point:

• data is unencrypted

• identity is unverified

• trust is zero

This is where HTTP originally lived.

When HTTP Was Enough (And Why It Stopped Being Enough)

Early web days were simple.

HTTP meant:

• plain text requests

• plain text responses

• anyone on the network could read or modify traffic

It worked because:

• websites were informational

• no logins

• no payments

• no personal data

But the moment we started sending:

• passwords

• cookies

• tokens

• personal data

HTTP became dangerous.

Anyone in between could:

• read your credentials

• hijack sessions

• modify responses

That’s when the web needed something below HTTP.

Enter SSL and TLS (The Invisible Guardians)

Here’s the part that confused me for years:

SSL and TLS are not web technologies.
They don’t know about URLs, controllers, or APIs.

They live below HTTP.

Their job is simple but critical:

“Make this connection private and trustworthy.”

Before a single HTTP request is sent, TLS steps in and asks:

• Who are you?

• Can I trust you?

• Can we talk securely?

• How should we encrypt our conversation?

Only if all answers are acceptable does HTTP get a chance to speak.

Step 3: The TLS Handshake (The Real Magic)

This is the moment most people never see.

The browser and server perform a TLS handshake.

Conceptually, this happens:

• Browser says: “Here are encryption methods I support”

• Server responds: “Here’s my certificate”

• Browser verifies:

  • Is this certificate valid?

  • Is it issued by a trusted authority?

  • Does it match the domain?

  • Is it expired or revoked?

If anything fails — everything stops.

No HTTP request.
No controller.
No API.

This is why HTTPS failures feel so dramatic.

Why Certificates Exist (And Why They Matter)

Certificates solve one critical problem:

How do you know the server you’re talking to is who it claims to be?

Without certificates:

• anyone could pretend to be example.com

• encryption alone wouldn’t help

• you’d encrypt data… to the wrong person

Certificates tie:

• domain name

• public key

• trusted authority

This is how browsers build trust without knowing your server personally.

Step 4: HTTPS Is Just HTTP With Armor

Here’s the big realization that clicked for me:

HTTPS is not a different protocol.
It’s HTTP inside a secure tunnel.

Once TLS finishes:

• HTTP requests are sent normally

• headers, cookies, payloads behave the same

• your application code sees no difference

The difference is:

• nobody else can read it

• nobody can alter it

• the server identity is verified

That’s it.

HTTPS doesn’t change how you write APIs.
It changes who can see and trust them.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Where Cookies, Sessions, and Auth Depend on This

This is where everything connects.

Cookies only work securely because:

• HTTPS protects them from being stolen

• TLS prevents session hijacking

• browsers enforce secure cookie rules

Without HTTPS:

• authentication falls apart

• sessions become vulnerable

• trust collapses

That’s why modern browsers now block sensitive features on HTTP.

Security moved from “nice to have” to “non-negotiable”.

Step 5: The Actual HTTP Request Finally Happens

Only now does the browser send:

GET / HTTP/1.1
Host: example.com

At this point:

• the connection is encrypted

• the server is trusted

• cookies are attached

• headers are intact

Your server code finally runs.

Controllers execute.
Middleware triggers.
Responses are generated.

But all of this sits on top of what already happened.

Why We Need All of This (The Simple Truth)

Each layer exists because the one above it assumes safety.

• HTTP assumes a reliable connection

• HTTPS assumes a trusted identity

• Cookies assume encryption

• Authentication assumes integrity

Remove one layer, and the rest become fragile.

This isn’t complexity for the sake of it.

It’s defense built from experience.

The Moment It Finally Made Sense to Me

The day this truly clicked was when I realized:

Application security doesn’t start in your code.
It starts before your code even runs.

HTTP, HTTPS, TLS, certificates —
they’re not optional add-ons.

They’re the foundation that makes modern applications possible.

Final Thought

👉 I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

Next time you type a URL and hit Enter, remember:

You’re not “opening a website”.

You’re:

• resolving identity

• negotiating trust

• agreeing on encryption

• establishing privacy

• and only then… exchanging data

All in a fraction of a second.

Once you see that,
words like HTTP, HTTPS, SSL, and TLS stop being confusing acronyms.

They become quiet guardians — doing their job so your application code can do its own.

And when you understand that, you stop taking the web for granted.

You start respecting it.

For a long time in my career, I believed report generation was the end of the story.

A user clicks Download Report.
The backend queries the database.
A file is generated.
The user gets it.

Done.

That mental model worked — until it didn’t.

The moment reports stopped being:

• small

• synchronous

• user-triggered

• single-consumer

Everything changed.

This blog is about that moment.
And about a flow that finally made sense to me:

.NET Application → Kafka → Ingestion Service → Shared Persistence / Blob Storage

At first, this flow looked unnecessarily complex.

Later, it felt inevitable.

When Reports Were Simple (and Why That Was Deceptive)

Early on, reports lived inside the application.

A controller would:

• fetch data

• format it

• write a file

• stream it back

It felt productive.

But over time, requirements grew quietly:

• reports became large

• generation took minutes

• users wanted async processing

• multiple teams wanted the same report

• reports needed to be stored, audited, reprocessed

Suddenly, “generate and return” wasn’t enough.

The First Cracks: Reports Are Not Requests

The first realization was subtle but important:

A report is not a request-response problem.

A report is:

• heavy

• long-running

• retryable

• replayable

• consumed by more than one party

Treating it like an HTTP request forces:

• timeouts

• retries from browsers

• duplicated work

• tight coupling between UI and backend

Once I accepted that, the architecture had to change.

Step 1: The .NET Application Stops “Doing” the Report

The biggest shift was this:

The .NET application stopped generating reports
and started declaring that a report should exist.

Instead of:

“Generate report now”

The app began to say:

“A report of type X, with parameters Y, is required.”

That declaration became an event, not a function call.

This is where Kafka entered the picture.

Why Kafka Was the Right Fit (Not a Queue, Not a Database)

At first glance, Kafka felt like overkill.

But then the nature of reports became clearer.

A report request:

• is a fact (“this report was requested”)

• may be processed later

• may be processed multiple times

• may be reprocessed if logic changes

• must be auditable

Kafka fits because it treats events as records of truth, not tasks to discard.

The .NET app publishes something like:

“ReportRequested”

And then… it’s done.

No waiting.
No heavy lifting.
No assumption about who processes it.

That separation alone reduced stress in the codebase.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

What the .NET Application Is Responsible For (and What It Is Not)

This part matters.

The .NET application:

• validates input

• determines report intent

• publishes the event to Kafka

• returns immediately to the user

It does not:

• fetch all report data

• format files

• write to storage

• worry about retries or failures

This was uncomfortable at first.

Letting go of control always is.

But it made the app lighter, faster, and easier to reason about.

Kafka’s Role: The Unbiased Recorder

Kafka does not care about reports.

That’s its strength.

Kafka simply records:

• report requested

• with parameters

• at this time

• by this source

It doesn’t:

• decide how to generate

• care how long it takes

• delete the message after consumption

That means:

• multiple ingestion services can read the same request

• failures don’t lose intent

• logic can evolve without re-triggering users

Kafka became the memory of intent.

Enter the Ingestion Service: Where Work Actually Happens

This is where the real work lives.

The ingestion service is not a controller.
Not a UI.
Not a database.

It’s a specialized worker whose only responsibility is:

“Turn report intent into report artifacts.”

It:

• consumes report events from Kafka

• fetches data from multiple sources

• applies business rules

• formats output (CSV, PDF, Excel, JSON)

• handles retries and partial failures

This service is allowed to be:

• slow

• CPU-heavy

• memory-heavy

Because it’s isolated.

Failures here don’t take down the user-facing app.

That isolation was the biggest architectural win.

Why Ingestion Is a Separate Service (and Not Just a Background Job)

At some point I asked:

“Why not just run this as a background job in the same app?”

The answer came from experience:

Because ingestion:

• scales differently

• fails differently

• evolves differently

• needs different observability

• has different SLAs

Once reports were critical, ingestion became a system, not a helper.

Separating it allowed:

• independent scaling

• independent deployments

• independent retry strategies

And that independence saved us more than once.

Shared Persistence: The Final Destination

After generation, reports need a home.

Not inside memory.
Not inside the ingestion service.
Not inside Kafka.

They need shared persistence.

This is where blob storage (or shared file systems) comes in.

Blob storage is ideal because:

• it’s durable

• it’s cheap

• it’s scalable

• it’s accessible by multiple consumers

• it decouples generation from consumption

Once a report is written to blob storage:

• users can download it later

• other services can process it

• analytics jobs can read it

• retention policies can apply

The report becomes a first-class artifact, not a temporary output.

The Flow, End to End (How It Feels in Practice)

Here’s how the system feels when designed this way:

1. User requests a report

2. .NET app validates and publishes intent to Kafka

3. Kafka records the request

4. Ingestion service consumes when ready

5. Report is generated asynchronously

6. Output is written to shared storage

7. Status updates can be published back if needed

No blocking.
No guessing.
No hidden coupling.

Just a clean flow of responsibility.

👉 I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

The Biggest Mental Shift This Flow Taught Me

This architecture taught me something important:

Systems work better when intent, processing, and storage are separate concerns.

Trying to collapse them into one layer feels simpler —
until scale, failures, and change arrive.

Then separation stops feeling like ceremony
and starts feeling like relief.

Why This Design Ages Well

This flow survives change because:

• report formats can change without UI changes

• ingestion logic can evolve independently

• storage policies can change without regeneration

• new consumers can read old events

• reprocessing is possible without user involvement

That’s not over-engineering.

That’s designing for time.

Final Thought

At first, this architecture feels indirect.

Why not just generate the report?

But once you live with it, you realize:

The complexity was always there.
You just moved it to the right places.

When the .NET app stops being a factory
and starts being a coordinator,
everything becomes calmer.

And calm systems are the ones that survive.

Every backend developer reaches this moment.

You’re designing a system.
Someone says:

“Let’s put Kafka.”
Another says: “Service Bus is enough.”
Someone else: “RabbitMQ is simpler.”
And a senior casually drops: “This is just ingestion.”

At this point, your brain doesn’t explode.
It freezes.

Because all of them:

• move messages

• decouple systems

• run in the background

• sound interchangeable

But they are not the same thing.

This blog will untangle them in a way you’ll never confuse again — using family metaphors, real system behavior, and .NET examples.

By the end, you’ll know:

• who belongs to the same family

• who are siblings vs cousins

• who are neighbors doing different jobs

• and which one YOU should pick — and why

First, the big picture (this is the anchor)

Before naming tools, understand this truth:

These systems exist to move data between systems without forcing them to talk directly.

That’s it.

But how they move data
when they move data
who remembers it
and who guarantees delivery
— that’s where they differ.

The “Family Tree” (mental model)

Let’s group them correctly.

The Big Family: Asynchronous Messaging & Streaming

Inside this family, we have two major tribes:

1. Message Brokers (Queues & Buses)

2. Event Streaming Platforms

And standing nearby (but not exactly family):

3. Ingestion Services

Now let’s meet each one properly.

Message Queue — The Grandparent Concept

A message queue is the idea, not a product.

It means:

• one system sends a message

• another system processes it later

• they don’t block each other

Key characteristics:

• message is consumed once

• after consumption, it’s usually gone

• focus is on task execution

Think:

“Do this job when you get time.”

Example scenario

• User uploads a file

• Backend enqueues “ProcessFile”

• Worker processes it in background

This idea existed long before Kafka or cloud.

In .NET, this concept shows up everywhere:

• Background workers

• Hosted services

• Queues

• Even Task.Run() is a tiny cousin of this idea

RabbitMQ — The Responsible Sibling

RabbitMQ is a classic message broker.

It lives and breathes message queue semantics.

What RabbitMQ is really good at

• Command-style messages

• Job processing

• Reliable delivery

• Acknowledgements

• Routing (fanout, topic, direct)

RabbitMQ thinks like this:

“I must make sure this message is handled — exactly once if possible.”

Mental model

RabbitMQ is like a post office:

• letters arrive

• post office routes them

• once delivered, letter is done

Simple .NET producer example (conceptual)

var factory = new ConnectionFactory { HostName = "localhost" };
using var connection = factory.CreateConnection();
using var channel = connection.CreateModel();

channel.QueueDeclare("orders");

var message = Encoding.UTF8.GetBytes("OrderCreated");

channel.BasicPublish(
    exchange: "",
    routingKey: "orders",
    body: message
);

Consumer

channel.BasicConsume("orders", autoAck: false, consumer);

RabbitMQ:

• expects consumers to ack

• retries if they fail

• cares deeply about delivery guarantees

Azure Service Bus — The Enterprise Cousin

Azure Service Bus is not just a queue.

It’s a managed enterprise messaging platform.

If RabbitMQ is a post office,
Service Bus is a corporate mailroom with rules.

What Service Bus adds

• Queues and Topics

• Dead-letter queues

• Scheduled messages

• Sessions (ordered processing)

• Transactions

• Azure-native security & scaling

It’s designed for:

• enterprise workflows

• business processes

• financial systems

• regulated environments

Mental model

“This message is part of a business process — treat it carefully.”

.NET example (simplified)

var client = new ServiceBusClient(connectionString);
var sender = client.CreateSender("orders");

await sender.SendMessageAsync(
    new ServiceBusMessage("OrderCreated")
);

Service Bus does not want to be fast above all.
It wants to be correct, secure, and auditable.

Kafka — The Loud, Fast, Memory-Rich Sibling

Kafka is not a message queue — even though people use it like one.

Kafka belongs to a different tribe.

Kafka is an event streaming platform.

How Kafka thinks

Kafka doesn’t ask:

“Did someone process this message?”

Kafka asks:

“I will record this fact forever. Others can read it when they want.”

Core differences (intuitively)

• Messages are not deleted after consumption

• Consumers track their own offset

• Multiple consumers can read the same data

• Data is replayable

• Order matters deeply

Kafka is like a public event log:

“These events happened. Deal with it.”

Example Kafka mindset

• UserCreated event

• OrderPlaced event

• PaymentFailed event

These are facts, not commands.

.NET producer (conceptual)

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using var producer = new ProducerBuilder<Null, string>(config).Build();

await producer.ProduceAsync(
    "orders",
    new Message<Null, string> { Value = "OrderPlaced" }
);

Kafka shines when:

• many systems need the same data

• replay is required

• analytics + services coexist

• throughput is massive

Ingestion Services — The Neighbours (Not Family)

Ingestion services are not brokers.

They are pipelines.

Their job:

• receive data

• transform/validate/enrich

• forward to storage or streams

Examples:

• log ingestion

• metrics ingestion

• telemetry pipelines

• ETL-lite services

Mental model

“I don’t store messages. I process flow.”

Often:

• Kafka feeds ingestion

• Ingestion feeds databases

• Ingestion feeds analytics

In .NET, ingestion often looks like:

• background services

• streaming processors

• consumers + transformers

How they relate (final clarity)

Are they twins?

❌ No.

Are they siblings?

✔️ RabbitMQ & Service Bus are siblings
✔️ Kafka is from the same family but different tribe

Same family?

✔️ Yes — asynchronous systems

Neighbours?

✔️ Ingestion services are neighbours — they use brokers, they are not brokers

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Other similar concepts in the .NET world

1️⃣ Background Workers

• IHostedService

• BackgroundService

• In-process queues

Good for:

• small workloads

• internal async tasks

Not good for:

• distributed systems

• scaling independently

2️⃣ Event Sourcing

• Often built on top of Kafka

• Events are source of truth

• Read models built separately

Kafka fits this naturally.

3️⃣ SignalR

• Not a queue

• Not streaming

• Real-time push to clients

Neighbour, not family.

4️⃣ Azure Event Grid

• Event notification system

• Fire-and-forget

• No replay

• No ordering guarantee

Good for:

• reacting to cloud events

• glue between services

How to choose (the honest guidance)

Ask one question:

“Am I sending a command or publishing a fact?”

If it’s a command:

• “Process this”

• “Send this email”

• “Charge this payment”

→ RabbitMQ / Service Bus

If it’s a fact:

• “Order was placed”

• “User logged in”

• “Payment failed”

→ Kafka

If it’s transformation flow:

• logs

• metrics

• telemetry

→ Ingestion service (often backed by Kafka)

The one-line memory trick

If you remember nothing else, remember this:

Queues ask: “Did you do it?”
Kafka says: “This happened.”

That sentence alone separates 80% of confusion.

👉 I’ve shared the JWT Authentication Boilerplate for ASP.NET Core (.NET 8)
(Instant download, production-ready, no fluff)

Final closing

Most teams don’t fail because they picked the “wrong” tool.

They fail because:

• they didn’t understand the nature of their data

• they treated facts like commands

• or commands like facts

Once you see these systems as roles in a family, not interchangeable tools, your designs become calm, intentional, and scalable.

And that’s when you stop being “the dev who knows Kafka”
and become the architect who knows why.

For most developers, the shift from junior to expert happens on an ordinary day, in front of an ordinary screen, staring at code they’ve written a hundred times before — and suddenly realizing:

“I don’t actually understand what’s happening here.”

That moment is uncomfortable.
But it’s also the beginning of something important.

Chapter 1: When “Learning .NET” Stops Feeling Like Progress

Early in your career, learning .NET feels exciting.

You learn:

• Controllers

• Services

• Dependency Injection

• async/await

• Entity Framework

Every new concept feels like a level unlocked.

Your code compiles.
Your APIs respond.
Your manager is happy.

But one day, something strange happens.

The code still works…
…but it no longer feels solid.

You deploy a change and something unrelated breaks.
A small feature takes days to debug.
Performance issues appear without obvious reason.

You realize you’re not stuck because you don’t know enough.

You’re stuck because you don’t see the system yet.

Chapter 2: The First Time async/await Betrays You

You remember the day you learned async/await.

It felt magical.
No more blocking.
Everything faster.

So you used it everywhere.

And then one day, production slows down.

Requests hang.
CPU is fine.
Memory looks okay.

Logs don’t scream.
They whisper.

You stare at the code and think:

“But I followed the rules…”

That’s the day async/await stops being syntax and starts becoming behavior.

You begin to imagine threads being released.
Continuations waiting quietly.
The thread pool slowly starving.

You stop asking:

“Where should I put await?”

And start asking:

“What is the runtime doing right now?”

That’s not learning anymore.
That’s experience being carved into your thinking.

Chapter 3: The God Service You Were Afraid to Touch

There’s always that one service.

Huge.
Scary.
Everyone avoids it.

You didn’t plan to create it.
You just kept adding “one more thing”.

Validation went there.
Business rules went there.
Email sending went there.

Now, every change feels risky.

You open the file and your instinct says:

“Close it before you break something.”

That fear teaches you something tutorials never did:

Responsibilities matter more than features.

You realize Dependency Injection wasn’t meant to make things flexible.
It was meant to make change survivable.

From that day on, you stop asking:

“Can I inject this?”

And start asking:

“Who should own this decision?”

That’s architecture entering your bloodstream.

Chapter 4: The Database Stops Feeling Like Storage

At first, the database was just there.

You queried it.
You updated rows.
You fixed bugs with WHERE clauses.

Until the day data was wrong.

Not slow.
Not missing.
Wrong.

Balances didn’t match.
Reports disagreed.
History couldn’t be trusted.

You realize something quietly terrifying:

Once data lies, nothing above it can be trusted.

That’s when you stop treating the database as storage
and start treating it as a contract with reality.

You think about:

• Invariants

• Ownership

• What should never change

• What must be traceable forever

You stop asking:

“How do I fetch this?”

And start asking:

“Why does this data exist like this?”

That’s not SQL knowledge.
That’s system thinking.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Chapter 5: Performance Stops Being About Speed

Junior you thought performance was about speed.

Loops.
Indexes.
Caching.

Expert you knows performance is about cost.

You’ve seen systems where:

• Caching made things worse

• Optimization hid deeper problems

• GC pauses killed throughput

You stop touching code first.

You sit back.
You measure.
You observe.

You ask:

• Is this IO or CPU?

• Why is this path hot?

• Can we reduce work instead of speeding it up?

You realize most performance issues were design decisions made long ago.

And suddenly, performance becomes calm instead of frantic.

Chapter 6: Errors Become Expected Guests, Not Surprises

There was a time when errors scared you.

Exceptions meant failure.
Failure meant panic.

Then you got paged at night.

And you learned the truth:

Failures are not accidents.
They are part of the system’s life.

You stop wrapping everything in try-catch.
You stop hiding errors.

You design failure paths.

You decide:

• What can retry

• What must stop

• What must alert humans

• What must fail loudly

You don’t ask:

“Did we handle the exception?”

You ask:

“Did we handle reality?”

That’s experience speaking.

Chapter 7: Debugging Becomes Storytelling

You no longer debug line by line.

You tell stories.

“What happened first?”
“What assumption failed?”
“How did state evolve?”

You replay the system in your head like a movie.

And most bugs suddenly make sense.

Because bugs were never random.

They were consequences.

Chapter 8: The Quiet Confidence of an Expert

One day, someone asks you a question in a meeting.

You don’t rush to answer.

You think.

You explain not just what you chose — but why.

You talk about trade-offs.
You mention risks.
You admit uncertainty.

And people listen.

Not because you know everything —
but because your thinking is visible.

That’s when you realize:

Expertise is not loud.
It’s calm.

Final Chapter: When Learning Turns Into Judgment

Experts don’t read more than juniors.
They reread basics with new eyes.

They don’t chase every new framework.
They deepen their understanding of fundamentals.

They don’t avoid mistakes.
They extract meaning from them.

And one day, without ceremony, you realize:

“.NET didn’t change.
I did.”

A Quiet Ending

If you’re a junior .NET developer reading this:

You don’t need to rush.
You don’t need to prove anything.
You don’t need to know everything.

Just pay attention.

To behavior.
To consequences.
To the moments when things break and you learn why.

That’s how learning turns into experience.

And that’s how developers become experts —
not overnight, but inevitably.

For content overview videos
https://www.youtube.com/@DotNetFullstackDev

If you’re a .NET developer, this blog is the missing bridge between:

• system-design interviews

• and real production code

We’ll design a SettleUp-like expense-splitting app, end to end:

• backend in .NET Core

• frontend in React

• database in SQL

And more importantly, we’ll explain why this stack makes sense — not because it’s popular, but because it fits the problem.

Why this tech stack (before we design anything)

Let’s answer the “why” upfront, because architecture always starts with constraints.

Why .NET Core for backend?

Because this problem needs:

• strong domain modeling

• clear separation of concerns

• predictable performance

• long-term maintainability

.NET Core gives us:

• clean layering (Domain, Application, Infrastructure)

• excellent async handling (important for APIs)

• strong typing for financial logic

• mature ecosystem for auth, logging, validation

This is business logic-heavy software, not a script.
.NET shines here.

Why React for frontend?

SettleUp-style apps are:

• highly interactive

• state-driven

• real-time feeling (balances update instantly)

• UI-heavy

React fits because:

• UI is just a function of state

• balance recalculations reflect immediately

• components map cleanly to domain concepts

• ecosystem supports charts, lists, notifications easily

This isn’t a static website.
It’s a living financial dashboard.

Why SQL database?

This is the most important choice.

Expense sharing is financial data.

That means:

• correctness > flexibility

• consistency > speed hacks

• auditability > schemaless freedom

SQL gives us:

• transactions

• referential integrity

• constraints

• predictable queries

• easy reconciliation

Ledger-style systems and relational databases are a natural fit.

Big picture: how the system is split (low-level view)

We split the system into three responsibilities:

1. Frontend (React)

   • collects user intent

   • displays derived data (balances, settlements)

2. Backend (.NET Core API)

   • enforces business rules

   • computes balances

   • guarantees correctness

3. Database (SQL)

   • stores immutable financial history

   • never lies

Frontend never calculates money.
Backend never trusts frontend numbers.
Database never stores “derived” values blindly.

This separation saves you from nightmares later.

Backend low-level design (.NET Core)

Let’s go layer by layer — the way you’d actually structure the solution.

1️⃣ API Layer (Controllers / Endpoints)

This layer does one job only:

• accept requests

• validate shape

• call application services

• return responses

It must not:

• calculate balances

• apply business rules

• talk directly to EF entities

Example responsibilities:

• POST /groups

• POST /expenses

• GET /groups/{id}/balances

• POST /settlements

Controllers should feel boring.
That’s a good sign.

2️⃣ Application Layer (Use-Cases)

This is where intent becomes action.

Each use-case represents:

• a business action

• a user intention

Examples:

• CreateExpense

• AddGroupMember

• RecordSettlement

• GetGroupBalances

• GetSimplifiedSettlements

Each use-case:

• validates business rules

• coordinates domain objects

• controls transactions

Think of this layer as:

“The orchestrator of business workflows”

3️⃣ Domain Layer (the heart of correctness)

This is the most important part of SettleUp.

Here you model:

• Expense

• Split

• Settlement

• Balance calculation rules

Key design principle:

Domain objects protect invariants.

For example:

• Total splits must equal total expense

• Settlement amount must be positive

• A user cannot owe themselves

• Currency mismatch must be explicit

If these rules leak outside the domain, bugs creep in silently.

4️⃣ Balance calculation service (core logic)

This deserves special attention.

You do not store balances permanently.

Instead:

• fetch all ledger entries (expenses + settlements)

• compute net balances per user

• simplify debts if required

This logic must be:

• deterministic

• testable

• independent of database concerns

This is where interviews are won and production bugs are avoided.

5️⃣ Infrastructure Layer (EF Core + SQL)

This layer:

• persists domain entities

• maps objects to tables

• manages transactions

Important design choice:

• expenses and settlements are append-only

• updates create new records or mark old ones as inactive

Why?
Because financial history must be explainable.

Never “rewrite” money.

Database low-level design (SQL mindset)

The database represents truth, not convenience.

Ledger-first storage

Instead of:

• storing “A owes B ₹500”

We store:

• who paid

• who owed

• who settled

• when

• how much

Balances are derived views, not stored facts.

This enables:

• recomputation

• audits

• debugging disputes

• future rule changes

Transactions everywhere

Any operation that:

• creates an expense

• records a settlement

must be wrapped in a transaction.

If one insert fails, nothing should persist.

Money hates partial success.

Constraints > Code checks

Use SQL constraints for:

• foreign keys

• non-negative amounts

• required relationships

This protects you even if:

• someone bypasses the API

• a bug slips into code

Frontend low-level design (React)

Now let’s move to the UI — where things feel complex.

State-driven UI (not imperative UI)

In React, the UI is:

a projection of state

That means:

• no manual DOM updates

• no recalculations in UI

• backend sends truth

• frontend renders truth

Example:

• user adds expense

• API returns updated balances

• React re-renders automatically

Frontend should never “guess” balances.

Component structure (domain-aligned)

Good React apps mirror backend concepts:

• GroupList

• GroupDetails

• ExpenseList

• AddExpenseForm

• BalanceSummary

• SettlementSuggestions

When frontend names match backend concepts:

• bugs reduce

• onboarding is faster

• communication improves

API contract discipline

Frontend talks to backend via:

• DTOs

• well-defined response models

Never expose EF entities directly.

Frontend should not depend on:

• internal IDs

• database structure

• backend refactoring

This decoupling is what allows change without pain.

Why this LLD scales (even if the app grows)

This design supports:

• thousands of expenses

• many groups

• multiple currencies

• future features (notifications, analytics)

Because:

• ledger is immutable

• business logic is centralized

• UI is stateless relative to calculations

• derived data is recalculated, not guessed

You don’t paint yourself into a corner.

Common mistakes this design avoids

Let’s be honest — these are mistakes many devs make:

• ❌ calculating balances in frontend

• ❌ storing “who owes whom” directly

• ❌ mixing EF entities in controllers

• ❌ updating past financial records

• ❌ trusting UI calculations

This design deliberately avoids all of them.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final takeaway (the one to remember)

If you remember only one thing from this blog, remember this:

SettleUp is not a CRUD app.
It’s a ledger-driven financial system with a friendly UI.

When you design it like a ledger:

• correctness becomes natural

• edge cases become manageable

• scaling becomes predictable

• interviews become easier

• production bugs reduce drastically

For content overview videos
https://www.youtube.com/@DotNetFullstackDev

You’ve already completed authentication (for example, JWT Bearer, Windows Auth, or Identity Provider).
Now the request reaches your ASP.NET Core API with a valid identity.

The next big question is:

“How does .NET Core check roles and decide whether to allow this request?”

This blog explains the exact pipeline: what happens after authentication succeeds, how roles are created and attached to the user, and how ASP.NET Core enforces role-based access.

1) The moment authentication completes: what you now have

After authentication succeeds, ASP.NET Core has one key thing:

HttpContext.User — a ClaimsPrincipal

Think of it as an in-memory “passport” for this request. It contains:

• Identity.IsAuthenticated = true

• A set of claims (name, email, subject id, tenant id, roles, etc.)

At this stage, no endpoint access decision is made yet.
Authentication only answers: “Who is this?”

2) Where roles live in ASP.NET Core: roles are just claims

In ASP.NET Core, a role is usually represented as a claim:

• claim type: ClaimTypes.Role (or "role" / "roles" depending on your token)

• claim value: "Admin", "Manager", "Auditor", etc.

So when people say “role-based authorization”, what they really mean is:

“Authorization based on a specific claim that represents role membership.”

This matters because your system might authenticate users correctly, but if roles are not mapped into the right claim type, [Authorize(Roles="...")] won’t work.

3) The request pipeline: where authorization happens

Here’s the flow in ASP.NET Core:

1. Request enters middleware pipeline

2. UseAuthentication() runs

   • builds HttpContext.User

3. Endpoint routing identifies which action is being called

4. UseAuthorization() runs

   • checks [Authorize], roles, policies

5. If allowed → controller action executes

6. If not allowed → 401 or 403 returned

The critical middleware order

app.UseRouting();

app.UseAuthentication();  // builds HttpContext.User (identity + claims)
app.UseAuthorization();   // enforces [Authorize] attributes

app.MapControllers();

If this order is wrong, role authorization will behave unpredictably.

4) Step-by-step: how ASP.NET decides “401 vs 403”

This is one of the most confusing parts for juniors.

401 Unauthorized

Means: user is not authenticated.

• No valid token

• Token expired

• Signature invalid

• Auth scheme not satisfied

403 Forbidden

Means: user is authenticated, but not authorized.

• Token is valid

• User is known

• But role requirement failed

Role-based authorization mostly leads to 403 (because authentication is already done).

5) Implementing role authorization using JWT (most common)

Step 1: Add JWT authentication

using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidateAudience = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ValidIssuer = "your-issuer",
            ValidAudience = "your-audience",
            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes("super-secret-key-1234567890"))
        };
    });

builder.Services.AddAuthorization();
builder.Services.AddControllers();

var app = builder.Build();

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();

app.MapControllers();
app.Run();

Authentication now creates HttpContext.User.

But role enforcement depends on the role claim being correct.

6) Step-by-step: ensuring roles are recognized correctly

JWT tokens might store roles in different claim types, such as:

• "role"

• "roles"

• "http://schemas.microsoft.com/ws/2008/06/identity/claims/role"

ASP.NET Core checks role claims using the identity’s RoleClaimType.

If your token uses "roles" but ASP.NET expects ClaimTypes.Role, you must map it.

Fix: set RoleClaimType

Subscribe now

options.TokenValidationParameters = new TokenValidationParameters
{
    // other validations...
    RoleClaimType = "roles"
};

Now [Authorize(Roles="Admin")] will work when the token contains:

"roles": ["Admin", "Auditor"]

7) Add role-based protection to endpoints

Controller example

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class ReportsController : ControllerBase
{
    [HttpGet("public")]
    public IActionResult Public() => Ok("Anyone can view this.");

    [Authorize] // authenticated users only
    [HttpGet("secure")]
    public IActionResult Secure()
        => Ok($"Hello {User.Identity?.Name}, you are authenticated.");

    [Authorize(Roles = "Admin")]
    [HttpGet("admin")]
    public IActionResult AdminOnly()
        => Ok("Admin access granted.");

    [Authorize(Roles = "Admin,Auditor")]
    [HttpGet("audit")]
    public IActionResult AdminOrAuditor()
        => Ok("Admin or Auditor access granted.");
}

What happens internally when calling /admin

• User is authenticated

• ASP.NET checks roles required: "Admin"

• It searches HttpContext.User.Claims for role claims

• If match found → allow

• If not → return 403 Forbidden

8) “But my user is authenticated and still gets 403” (common reasons)

Reason A: Role claim type mismatch

Token contains "roles" but your system expects ClaimTypes.Role.

Fix using RoleClaimType.

Reason B: Roles are nested or formatted differently

Token might store roles as a single string:

"roles": "Admin"

Or as:

"roles": "Admin,Auditor"

ASP.NET expects multiple claims or an array-like representation.
If not, you may need transformation.

Reason C: Roles are not present at all

Authentication succeeded, but roles weren’t included in the token.

This is common when:

• IdP config doesn’t emit roles

• scopes are missing

• claims mapping is not configured

9) Role transformation (when you want to enrich roles post-auth)

Sometimes you authenticate with a token, but roles are stored in your database.

In that case, after authentication succeeds, you can add roles via claims transformation.

Example: Add roles after authentication

using System.Security.Claims;
using Microsoft.AspNetCore.Authentication;

public class RoleClaimsTransformation : IClaimsTransformation
{
    public Task<ClaimsPrincipal> TransformAsync(ClaimsPrincipal principal)
    {
        var identity = (ClaimsIdentity)principal.Identity!;

        // Example: add a role dynamically (replace with DB lookup)
        if (!identity.HasClaim(c => c.Type == ClaimTypes.Role && c.Value == "Admin"))
        {
            identity.AddClaim(new Claim(ClaimTypes.Role, "Admin"));
        }

        return Task.FromResult(principal);
    }
}

Register it:

builder.Services.AddScoped<IClaimsTransformation, RoleClaimsTransformation>();

Now roles can be attached after authentication and then authorization will evaluate them.

This is a clean pattern when your IdP doesn’t provide roles.

10) The clean mental model for juniors

After authentication:

• HttpContext.User exists

• It contains claims

• Role checks are claim checks

• [Authorize(Roles="X")] is evaluated in UseAuthorization()

• 401 means “no identity”

• 403 means “identity exists, but role mismatch”

If you keep this picture, role-based authorization becomes predictable.

Subscribe now

Share DotNet Full Stack Dev

Leave a comment

For content overview videos
https://www.youtube.com/@DotNetFullstackDev

Every developer has written this line thousands of times:

SELECT * FROM Orders WHERE OrderId = 101;

You press Execute.
A result appears.
And that’s it… right?

Not really.

Behind that single query, SQL Server runs an entire internal pipeline involving:

• parsing

• optimization

• compilation

• memory management

• locking

• execution engines

• storage engines

• buffer pools

• transaction logs

This article takes you inside SQL Server, step by step, to understand what really happens from the moment you hit Execute until rows are returned.

The Journey Begins: Client Sends the Query

When you execute a query from:

• SQL Server Management Studio (SSMS)

• an application (C#, Java, Python)

• a reporting tool

The query does not directly hit the database files.

First, it travels over the network using the TDS protocol (Tabular Data Stream).

What SQL Server receives

SQL Server receives:

• the SQL text

• session information

• login context

• transaction settings

• language and date formats

At this point, SQL Server has only text, not an execution plan.

Parsing: “Is This SQL Even Valid?”

The first internal component that touches your query is the Parser.

What parsing means

Parsing answers basic questions:

• Is the SQL syntax correct?

• Are keywords valid?

• Are parentheses balanced?

• Is the query structure meaningful?

If you write:

SELEC * FROM Orders

Parsing fails immediately.

Important clarification

Parsing does not:

• check if tables exist

• check indexes

• check performance

It only validates grammar.

If parsing succeeds, SQL Server moves to the next stage.

Binding (Algebrizer): “What Does This Query Refer To?”

Now SQL Server needs to understand what your query is talking about.

This step is often called:

• Binding

• or Algebrization

What happens here

SQL Server:

• Resolves table names

• Resolves column names

• Checks schemas

• Validates data types

• Verifies permissions

• Expands * into actual columns

For example:

Subscribe now

SELECT * FROM Orders

Becomes:

SELECT OrderId, OrderDate, CustomerId, TotalAmount FROM dbo.Orders

Now SQL Server knows exactly what objects are involved.

If a table or column doesn’t exist, this is where the error is thrown.

Query Optimization: “What Is the Fastest Way to Get the Data?”

This is the most intelligent and expensive part of query processing.

SQL Server now hands the query to the Query Optimizer.

What the optimizer does

The optimizer asks:

“There are many ways to execute this query.
Which one is the cheapest?”

Here, cheap means:

• less CPU

• fewer reads

• less memory

• faster execution

How SQL Server thinks (important mental model)

SQL Server does not run your query and then see how fast it is.

Instead, it:

1. Imagines many execution plans

2. Estimates the cost of each plan

3. Chooses the lowest-cost plan

This is based on:

• table statistics

• index statistics

• data distribution

• available indexes

• estimated row counts

Example

For this query:

SELECT * FROM Orders WHERE OrderId = 101;

Possible plans include:

• Full table scan

• Clustered index seek

• Non-clustered index seek + key lookup

The optimizer evaluates all options and picks one.

Plan Cache: “Have I Seen This Query Before?”

Before generating a new plan, SQL Server checks the Plan Cache.

Why this matters

Generating an execution plan is expensive.

So SQL Server asks:

“Do I already have a plan for this exact query?”

If yes:

• It reuses the existing plan

• Saves time and CPU

If no:

• A new plan is compiled

• Stored in the plan cache for future use

This is why parameterization and query consistency matter so much for performance.

Compilation: Turning Plan into Executable Instructions

Once the optimizer selects the best plan, SQL Server compiles it.

Compilation produces:

• an execution plan

• physical operators (Index Seek, Scan, Hash Join, Nested Loop)

• memory grant estimates

• execution order

At this point:

• SQL text → logical plan → physical plan

• SQL Server knows exactly how it will fetch the data

Still, no data has been read yet.

Execution Begins: The Execution Engine Takes Over

Now the query moves from the Relational Engine to the Storage Engine.

Important distinction

• Relational Engine: decides what to do

• Storage Engine: actually fetches data from disk/memory

Execution now starts operator by operator.

Buffer Pool: “Do I Already Have This Data in Memory?”

Before touching disk, SQL Server checks the Buffer Pool.

Buffer Pool explained simply

The buffer pool is SQL Server’s in-memory cache of data pages.

When SQL Server needs a data page:

1. Check buffer pool

2. If found → memory read (very fast)

3. If not found → disk read (slow)

This is why:

• memory is critical for SQL Server

• repeated queries get faster

• cold cache vs warm cache matters

Storage Engine: Reading Data Pages

If data is not in memory:

• SQL Server reads 8KB pages from disk

• Pages are loaded into the buffer pool

• Data is returned to the execution engine

SQL Server never reads individual rows.
It always works in pages.

Locks & Transactions: Keeping Data Safe

While reading or modifying data, SQL Server must ensure consistency.

What SQL Server does

• Acquires locks (shared, exclusive, intent)

• Honors transaction isolation level

• Prevents dirty reads or conflicts (based on settings)

Even a simple SELECT may acquire locks depending on:

• isolation level

• indexes used

• execution plan

Row Processing & Operator Execution

Now SQL Server executes the plan step by step.

For example:

• Index Seek → retrieves matching rows

• Filter → applies WHERE clause

• Join → combines rows from multiple tables

• Sort → orders results

• Aggregate → computes SUM, COUNT, etc.

Each operator:

• pulls rows from the previous operator

• processes them

• passes results upward

This happens row by row, not all at once.

Returning Results to the Client

As rows are produced:

• SQL Server streams them back to the client

• It does not wait for the entire result set

That’s why:

• you may start seeing rows early

• large queries return results gradually

The client displays the data as it arrives.

Cleanup: Memory, Locks, and Execution Context

Once execution finishes:

• locks are released

• memory grants are freed

• execution context is cleaned

• plan remains in cache (if eligible)

The query lifecycle is complete.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final Mental Model (Very Important)

When you execute a SQL query, SQL Server does far more than “run it”.

It:

1. Parses the SQL

2. Understands objects and permissions

3. Optimizes execution strategy

4. Reuses or compiles plans

5. Checks memory before disk

6. Reads data pages

7. Manages locks and transactions

8. Executes operators

9. Streams results back

Once you understand this internal pipeline:

• execution plans make sense

• indexing decisions become logical

• performance tuning becomes systematic

• SQL Server feels predictable, not magical

Subscribe now

Leave a comment

Share DotNet Full Stack Dev

“How does my frontend, mobile app, or another backend actually talk to this server?”

Very soon, you hear words like REST, gRPC, and SignalR being used casually in meetings, tutorials, and job descriptions. People say things like:

• “Expose a REST API”

• “Use gRPC for internal services”

• “SignalR will handle real-time updates”

For someone new, these sound like advanced, unrelated technologies.
But in reality, they are simply different ways of solving the same core problem — communication.

This article explains what these are, why they exist, why only these three dominate most backend systems, and whether other options exist, all in a slow, story-like manner.

First, the core idea (before protocols)

Before we talk about REST, gRPC, or SignalR, we must understand one foundational truth of backend systems.

A backend is rarely alone.

It almost always needs to communicate with something else:

• a web browser

• a mobile application

• another backend service

• a dashboard

• an external system

These systems are:

• running on different machines

• connected through networks

• possibly written in different programming languages

• possibly maintained by different teams

Because of this separation, they cannot just call each other’s methods directly like normal code inside the same application.

They need a defined way to talk.

That defined way is what we call a communication protocol.

What is a “protocol” in simple terms?

A protocol is not a library or a framework.
It is simply an agreement.

It answers questions like:

• How does a message start?

• How does it end?

• What format is used?

• Who speaks first?

• How does the receiver know what the sender wants?

Real-world analogy

Think about making a phone call:

• You say “Hello” first

• You wait for a response

• You take turns speaking

• You say “Bye” before hanging up

If one person suddenly starts shouting random words, communication breaks.

Backend communication works the same way.
Both sides must follow the same protocol rules.

REST — The Most Popular and Simplest One

REST is like sending letters by post

REST is the most widely used backend communication style, mainly because it is simple, familiar, and built on top of HTTP, which the entire internet already understands.

The REST model follows a very straightforward pattern:

• Client sends a request

• Server processes it

• Server sends a response

• Connection ends

Each interaction is self-contained.

How REST works (no jargon)

When a frontend calls a REST API:

1. It sends an HTTP request (GET, POST, PUT, DELETE)

2. The request contains a URL and optional data

3. The backend reads the request

4. The backend performs some logic

5. The backend sends back a response

6. The interaction ends

There is no memory of previous requests unless you explicitly build it.

Why REST became so popular

REST didn’t invent new technology.
It simply used what already existed:

• HTTP

• URLs

• browsers

• text-based formats like JSON

Because of this:

• developers could test APIs in a browser

• debugging was easy

• logs were readable

• tools like Postman worked instantly

This accessibility made REST explode in popularity.

When REST is perfect

REST shines when:

• you are building CRUD-based applications

• you expose public APIs

• readability and simplicity matter

• clients vary (web, mobile, third-party)

It is often the first protocol developers learn, and for good reason.

REST limitations

REST is not ideal for everything.

Because it is request–response:

• the client must keep asking for updates

• the server cannot push data by itself

• frequent polling increases load

• JSON payloads can become large

As systems became more complex and performance-sensitive, developers started looking for alternatives.

gRPC — When Performance and Efficiency Matter

gRPC is like a high-speed bullet train

gRPC was designed to solve problems REST struggles with, especially in backend-to-backend communication.

Instead of sending readable text like JSON, gRPC uses binary data, which is:

• smaller

• faster to parse

• more efficient over the network

Subscribe now

How gRPC works (simple view)

In gRPC:

• communication still happens over HTTP

• but it uses HTTP/2 instead of HTTP/1.1

• data is defined using strict contracts (.proto files)

• both client and server generate code from those contracts

This removes ambiguity and reduces errors.

Real-world analogy

REST is like writing letters that anyone can read.
gRPC is like sending compressed files between machines.

Humans prefer letters.
Machines prefer compressed files.

Why companies use gRPC

Companies choose gRPC when:

• performance is critical

• many services talk to each other

• data models must be consistent

• network efficiency matters

It is extremely common in:

• microservices

• internal service communication

• cloud-native systems

When gRPC is ideal

gRPC works best when:

• clients are known and controlled

• both sides are backend services

• low latency is important

• high throughput is required

Why gRPC is not everywhere

Despite its power, gRPC has drawbacks:

• harder to debug manually

• not human-readable

• not naturally supported by browsers

• learning curve is steeper

This is why REST remains dominant for public-facing APIs.

SignalR — When Backend Talks First

SignalR is like a live phone call

REST and gRPC share one limitation:

the client must initiate every request

But many modern applications need:

• instant notifications

• live updates

• continuous data streams

SignalR exists to solve this exact problem.

How SignalR works (simple)

With SignalR:

• the client opens a long-lived connection

• the connection stays open

• the server can send data anytime

• the client doesn’t need to ask repeatedly

This enables real-time communication.

Real-world analogy

REST is like checking your phone again and again for messages.
SignalR is like receiving a notification immediately when something happens.

Why SignalR exists

SignalR avoids:

• unnecessary polling

• wasted network traffic

• delayed updates

It creates a smoother user experience.

When SignalR is perfect

SignalR is ideal for:

• chat applications

• live dashboards

• notifications

• collaborative apps

• real-time monitoring systems

SignalR limitations

SignalR keeps connections open:

• which consumes server resources

• requires careful scaling

• is unnecessary for simple APIs

That’s why it complements REST and gRPC instead of replacing them.

Do other backend communication options exist?

Yes — but they are usually specialized tools, not general-purpose solutions.

GraphQL focuses on flexible queries.
WebSockets provide raw real-time communication.
Message queues enable asynchronous event processing.
SOAP exists mainly for legacy systems.

Each solves a specific problem, but none replace all three major protocols.

Why REST, gRPC, and SignalR dominate

These three dominate because together they cover almost every communication scenario:

• REST handles simple request–response APIs

• gRPC handles fast internal communication

• SignalR handles real-time updates

They are not competitors — they are complementary.

A simple mental map (very important)

Choosing a protocol becomes easy when you ask the right question:

• “Do I need simple request–response?” → REST

• “Do I need fast service-to-service calls?” → gRPC

• “Do I need real-time server push?” → SignalR

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final takeaway (for beginners)

Backend communication is not about picking the “best” protocol.

It’s about understanding what kind of conversation your systems need to have.

REST, gRPC, and SignalR exist because:

• backend systems evolved

• performance needs changed

• user expectations grew

Once you understand why they exist, choosing between them becomes natural.

Subscribe now

Share DotNet Full Stack Dev

Leave a comment

For content overview videos
https://www.youtube.com/@DotNetFullstackDev

Modern .NET uses one of the world’s most advanced memory-management engines — the Generational Garbage Collector.
But despite how powerful it is, many developers still feel unsure about:

• What exactly happens inside the GC?

• Why does .NET divide memory into Generations?

• How does the GC decide when to run?

• What does “promotion to Gen1/Gen2” really mean?

• When does Full GC happen?

The best way to understand this is to visualize the GC as a flow, not a single event.

Below, we walk through each step of the Generational GC process.

🧩 Step 1 — Object Allocated in Gen0

“Every new object starts its life in the nursery.”

When your application executes:

var customer = new Customer();

.NET allocates memory for this new object in the Gen0 heap, which is designed for:

• fast memory allocation

• short-lived objects

• minimal overhead

Think of Gen0 as a nursery — a space created for newborn objects.

Most objects in typical applications die young. For example:

• temporary strings

• LINQ intermediate objects

• short-lived DTOs

• loop variables

• objects used inside methods only once

Because these objects are created and destroyed quickly, Gen0 is optimized to handle them efficiently.

🧩 Step 2 — Gen0 Threshold Exceeded

“The nursery fills up.”

Gen0 has a small memory size because:

• small memory = fast cleanup

• frequent collection = efficient space recovery

As your program runs, more and more objects are created:

new X()
new Y()
new Z()

Once Gen0 becomes full, .NET must take action.

This is when the runtime says:

“Gen0 is full — time to clean up!”

This triggers the next step.

🧩 Step 3 — Gen0 Collection

“The GC quickly removes objects that are no longer needed.”

A Gen0 GC is the fastest type of garbage collection.

How does GC know what to delete?

• It pauses the application very briefly

• Checks which objects are still reachable (alive)

• Removes the ones that are no longer needed

All objects that survive this step get promoted to Gen1.

Why?
Because if an object survived a cleanup, it may be longer-lived.

🧩 Step 4 — Surviving Objects Promoted to Gen1

“If you survived the nursery cleanup, you move to the next room.”

Objects that weren’t cleaned up in Gen0 are considered more likely to live longer.

So .NET moves them to Gen1, which is a middle ground:

• bigger space

• collects less often

• handles objects with moderate lifetimes

Subscribe now

Typical Gen1 survivors include:

• objects reused across multiple method calls

• service instances

• collections that grow during processing

• cached objects

Think of Gen1 as a teenager stage — not newborn, not adult yet.

🧩 Step 5 — Gen1 Collection

“If Gen1 starts filling up, GC cleans it too.”

Eventually, Gen1 also fills.

When that happens, .NET triggers a Gen1 collection, which:

• cleans up dead Gen1 objects

• promotes surviving objects to Gen2

A Gen1 collection is slightly slower than Gen0, but still efficient.

This step only happens when needed — i.e., when Gen1 runs low on memory.

🧩 Step 6 — Long-Lived Objects Promoted to Gen2

“Any object that survives long enough becomes a long-term resident.”

Objects that survive Gen0 and Gen1 collections graduate to Gen2, the adult generation.

Gen2 contains:

• application-wide services

• large internal object graphs

• caches

• static data

• objects that stay alive for the lifetime of the app

Gen2 collection is rare and expensive, so .NET avoids touching it frequently.

Think of Gen2 as long-term storage.

🧩 Step 7 — Full GC (Triggered Only When Needed)

“When memory pressure becomes high, GC cleans everything.”

A Full GC is the most expensive type of garbage collection.
It includes:

• Gen0 cleanup

• Gen1 cleanup

• Gen2 cleanup

• LOH (Large Object Heap) cleanup

Full GC happens only under conditions like:

• very high memory usage

• large object allocations

• low available physical memory

• explicit calls like GC.Collect() (not recommended)

Full GC introduces a longer pause because the GC checks the entire managed heap.

.NET tries hard to avoid this, which is why generations exist — to optimize GC and minimize full collections.

🧩 Step 8 — Compaction & Finalization

“GC organizes memory and finalizes special objects.”

After deleting unused objects, .NET may:

1. Compact memory

This reduces fragmentation by:

• shifting objects together

• freeing large continuous memory blocks

Compaction is vital because memory fragmentation can crash your app or slow down JIT allocations.

2. Run finalizers (if any)

Objects with destructors (~MyClass()) are queued for special cleanup.

Finalizers delay collection, so avoid them unless necessary.

🧩 Step 9 — Execution Resumes

“The app continues running normally.”

Once GC completes its work:

• memory is clean

• generations are reorganized

• objects are promoted or removed

• memory pressure reduced

The application resumes exactly where it paused.

Modern .NET uses Background GC, meaning much of the GC work happens concurrently without stopping your program.

This gives you:

• faster apps

• smoother performance

• less noticeable GC pauses

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

⭐ Putting It All Together

The Generational GC system follows a beautiful, logical flow:

1. New objects go to Gen0

2. Gen0 fills → Collection happens

3. Survivors move to Gen1

4. Gen1 fills → Collection happens

5. Survivors move to Gen2

6. Gen2 grows slowly → Full GC if needed

7. Memory compacted & finalized

8. Execution resumes with fresh memory

This is not just cleanup — it is a performance strategy:

• Fast cleaning of short-lived objects

• Rare cleaning of long-lived objects

• Reduced pauses

• Smarter memory use

• Predictable performance

This is why .NET’s GC is one of the best in the industry.

Subscribe now

Leave a comment

Share DotNet Full Stack Dev

For content overview videos
https://www.youtube.com/@DotNetFullstackDev

Kerberos is everywhere in large enterprises—finance, telecom, healthcare, pharma, analytics, and government networks.
If you’re building Scala-based microservices using:

• Akka HTTP

• Play Framework

• Apache Spark

• Kafka clients

• Hadoop/YARN

• Distributed clusters

…you will almost certainly encounter Kerberos.

But here’s the real problem:

Most Kerberos explanations feel like reading an ancient spellbook.
Terms like TGT, SPN, KDC, AS-REQ, ticket enciphered — overwhelm new developers immediately.

So let’s rewrite this story in a way that makes sense to a real programmer.

This blog walks you through:

1. What Kerberos is (in simple human language)

2. How Kerberos works in a real enterprise

3. How Scala apps authenticate using Kerberos

4. Actual Scala code using JAAS + Kerberos principals

5. How Akka HTTP or Spark adopts Kerberos automatically

By the end, you’ll feel confident building Kerberized Scala services.

1. Kerberos in One Simple Analogy (No Security Jargon Needed)

Imagine a big corporate building.

You don’t type your password at every door.
Instead:

• Security desk verifies you once using your ID card

• They give you access stickers (tickets)

• Each sticker lets you enter certain rooms

• Rooms trust the sticker because security issued it

This is exactly how Kerberos works.

• Security desk = Domain Controller (KDC)

• Your corporate login = Kerberos principal

• Access stickers = Kerberos tickets

• Rooms = Scala services like Spark, Akka, Kafka

Your Scala application is a “room” that only accepts visitors with valid stickers.

2. How Kerberos Works — The Scala Developer’s Version

Kerberos uses two kinds of tickets:

TGT (Ticket Granting Ticket)

Issued when you authenticate the first time.

Service Ticket

Issued for each specific service (e.g., Kafka, Spark).

High-level flow:

1. User logs into the system → gets TGT

2. Scala application requests access to a service (e.g., Kafka broker)

3. TGT → exchanged for Service Ticket

4. Scala client sends the service ticket in every request

5. Service (broker/web API) validates ticket against KDC keys

No passwords go over the network.

3. Why Kerberos Matters in the Scala World

Scala is heavily used in:

• Big-data platforms

• Event streaming systems

• Distributed computing clusters

• High-security enterprise environments

These environments often run:

• Hadoop / HDFS (Kerberos mandatory)

• Spark on YARN

• Kafka clusters with SASL/Kerberos

• Akka microservices inside corporate domains

Kerberos is the first-class citizen of security in JVM ecosystems.

4. JAAS — The Bridge Between Scala and Kerberos

Scala uses the Java Authentication and Authorization Service (JAAS) under the hood.

Kerberos in Scala usually requires a JAAS config file:

jaas.conf

Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    useTicketCache=false
    keyTab=”/opt/keys/scala-service.keytab”
    principal=”scala-service@YOURDOMAIN.COM”;
};

This file tells the JVM:

• Which principal to use

• Where the keytab file is

• Whether to use ticket cache

5. Authenticating with Kerberos in Pure Scala

Scala code to login using JAAS + Kerberos:

import javax.security.auth.login.{AppConfigurationEntry, Configuration, LoginContext}

object KerberosAuthDemo {

  def main(args: Array[String]): Unit = {

    // Load the JAAS configuration
    System.setProperty(”java.security.auth.login.config”, “/opt/jaas.conf”)
    System.setProperty(”java.security.krb5.conf”, “/etc/krb5.conf”)

    try {
      val loginContext = new LoginContext(”Client”)
      loginContext.login()

      println(”✔ Kerberos authentication successful!”)
      println(”Authenticated principal: “ +
        loginContext.getSubject.getPrincipals)
    }
    catch {
      case ex: Exception =>
        println(”❌ Kerberos authentication failed: “ + ex.getMessage)
    }
  }
}

What this code does:

• Reads JAAS file

• Fetches the principal/keytab

• Asks Kerberos KDC for a TGT

• Logs the principal

If authentication works, your Scala process now has a valid ticket.

6. Kerberos + Akka HTTP (Server-Side Authentication)

Akka HTTP can sit behind:

• Apache/Nginx with Kerberos enabled

• Or Windows AD domain reverse proxy

Kerberos usually terminates at the reverse proxy, which then forwards identity headers to Akka.

Example Akka route:

val route =
  extractRequest { req =>
    val user = req.headers.find(_.name == “X-Authenticated-User”)
      .map(_.value)
      .getOrElse(”anonymous”)

    complete(s”Hello $user! You were authenticated by Kerberos.”)
  }

Your reverse proxy sets:

X-Authenticated-User: DOMAIN\username

7. Kerberos + Kafka (Very Common in Scala)

If your Scala service connects to Kafka:

val props = new java.util.Properties()

props.put(”security.protocol”, “SASL_PLAINTEXT”)
props.put(”sasl.mechanism”, “GSSAPI”)
props.put(”sasl.kerberos.service.name”, “kafka”)
props.put(”bootstrap.servers”, “kafka1:9092”)
props.put(”group.id”, “my-scala-group”)

val consumer = new KafkaConsumer[String, String](props)

Kafka automatically uses JAAS + keytab to authenticate.

8. Kerberos + Apache Spark (Another Huge Use Case)

In secured Hadoop clusters:

kinit -kt spark.keytab spark-user@REALM.COM
spark-submit --principal spark-user --keytab spark.keytab job.jar

Spark obtains a TGT → then uses that to interact with:

• YARN

• HDFS

• Hive

• Kafka

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

9. Summary — Kerberos Isn’t Hard. It Was Just Never Explained Simply.

Kerberos feels complex because it mixes:

• network authentication

• cryptography

• AD configuration

• JVM internals

But at its heart, it’s simple:

“Authenticate once → receive tickets → present tickets to Scala services.”

Scala integrates beautifully with Kerberos because:

• JVM has native support

• JAAS provides a simple configuration model

• Frameworks like Spark, Kafka, and Akka already support it

Today every project — whether it’s a .NET system, a BI pipeline, an ETL workflow, or a simple reporting job — eventually asks the same question:

“Should I output this data as CSV, JSON, or Excel?”

At first, it feels simple.
But when you start preparing the actual files, suddenly:

• Your CSV breaks because someone typed a comma inside a comment

• Your JSON fails because of a missing bracket

• Your Excel file becomes 45MB and refuses to open on someone’s laptop

• Your API consumer says “we need nested objects, not flat data”

• Your business stakeholder says “add colors and formulas inside Excel”

This blog will remove all that confusion.

Let’s break down the key differentiating factors, real-time use cases, and best practices for preparing CSV, JSON, and Excel — the three most used data formats in the world.

First Step: Understand What These File Formats Actually Represent

Before comparing, understand the nature of the formats.

CSV = Flat data

• Rows + columns

• No hierarchy

• Good for large datasets

• Very lightweight

• Supported everywhere

JSON = Structured + hierarchical data

• Perfect for APIs

• Supports nested objects, arrays

• Human + machine friendly

• Ideal for configuration or complex data structures

Excel = Rich data container

• Tables

• Formatting

• Charts

• Formulas

• Multiple sheets

• Business-friendly

Think of it like this:

FormatReal-world analogyCSVA notebook with only lines — pure raw dataJSONA container with sections, boxes, and labelsExcelA decorated project file with charts, colors, and formulas

Now let’s go deeper.

When Preparing CSV Files — Key Things to Consider

CSV is deceptively simple.

But preparing a CSV correctly requires precision, otherwise one wrong comma breaks the entire dataset.

1. Always define your delimiter (comma, semicolon, tab)

Default is comma (,).
But sometimes fields themselves contain commas:

“John, Alex”, 29, “India”

If you do not quote such fields, your CSV becomes invalid.

Many European countries prefer semicolon (;) because comma is used as a decimal separator.

2. Escape or quote values that contain special characters

Special characters include:

• commas

• newlines

• tabs

• quotes

Follow correct quoting:

“Hello, World”
“This is a “”quoted”“ value”
“Line1
Line2”

3. Keep column order consistent

Never reorder columns between exports unless absolutely necessary.

Systems downstream may break.

4. Avoid merging data types in the same column

Bad:

Name, Value
Age, 30
Price, $15

Good:

FieldName, FieldValue, FieldType
Age, 30, Integer
Price, 15, Currency

5. Avoid leading zeros if values represent IDs

Excel automatically removes leading zeros (e.g., 00123 → 123).

Workaround:

• Quote them “00123”

• Prepend ‘ (apostrophe)

6. Choose UTF-8 encoding

Prevent issues with:

• special characters

• Indian regional languages

• emojis

• accented names

7. Don’t include formulas

CSV is flat text.
No formulas, no formatting.
If you need formulas → Excel.

When Preparing JSON Files — Key Things to Consider

JSON supports complex, deeply nested structures.

But that power brings its own challenges.

1. Valid JSON structure is mandatory

One missing } will break everything.

Use online validators or automated serializers.

2. Be careful with data types

JSON allows:

• numbers

• strings

• arrays

• objects

• booleans

• null

All numbers default to float / double type — watch out for precision (especially with money).

3. Decide between compact vs pretty JSON

Compact:

{”user”:”John”,”age”:29}

Pretty formatted:

{
  “user”: “John”,
  “age”: 29
}

Pretty printing is good for humans;
compact is good for APIs and file transfer.

✔ 4. Use consistent naming style

• camelCase

• PascalCase

• snake_case

Example:

{
  “firstName”: “John”,
  “lastName”: “Alex”
}

5. Keep boolean values boolean, not strings

Bad:

“isActive”: “true”

Good:

“isActive”: true

6. Large JSON → consider splitting

If your JSON is larger than 5MB:

• Break into smaller JSON lines → NDJSON (“newline delimited JSON”)

• Use streaming serializers

7. JSON doesn’t support comments

Never write:

{
   // this is a comment
}

It breaks parsing.

When Preparing Excel Files — Key Things to Consider

Excel is powerful, but tricky.

It’s the only format that business teams often request because:

• They want filters

• They want pivot tables

• They want colors

• They want formulas

• They want multiple sheets

But as a developer, Excel is the easiest format to accidentally misuse.

1. Avoid more than 1 million rows

Excel’s row limit is:

• 1,048,576 rows

• 16,384 columns

If you exceed → use CSV instead.

2. Avoid embedding large images inside sheets

This increases file size like crazy.

3. Prefer simple formats rather than heavy styling

Excessive styling = slow opening.

4. If your Excel will be consumed by a program, avoid:

• merged cells

• formulas

• conditional formatting

• hidden rows

• auto filters

These break automated parsing.

5. If you DO need formulas — place them carefully

Example:

=SUM(A2:A100)
=IF(B2>10, “High”,”Low”)

Always document formulas in a separate sheet.

6. Use freeze panes for readability

Useful for files that business teams use.

7. Use Excel libraries correctly in .NET

Recommended libraries:

• EPPlus

• ClosedXML

• NPOI

Each has pros and cons depending on file size & complexity.

8. Watch out for Excel’s weird auto-conversion

Excel converts:

• “1-2” into a date

• “0012” into “12”

• “TRUE” into boolean

• “1.5E5” into exponential

• long numbers like credit-card digits into scientific notation

If the content is sensitive → store as text intentionally.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final Comparison — Choosing the Right Format (Simple Mental Model)

✔ Choose CSV when:

• You have large or flat data

• You don’t need formatting

• You want fastest export & import

• You want lightweight & universal

✔ Choose JSON when:

• You need nested structure

• You are interacting with APIs

• You need config files or metadata

• You need both human & machine readability

✔ Choose Excel when:

• Business users are the target

• You need formulas/charts

• You need multiple sheets

• You want filters, pivots, styling

The Ultimate One-Line Rules

CSV:

Use when speed matters. Flat data only.

JSON:

Use when structure matters. Machines + APIs.

Excel:

Use when humans matter. Reporting + visualization.

Closing Note

Choosing the right file format is a subtle but crucial decision in every project.
Knowing the strengths, limits, and preparation rules of CSV, JSON, and Excel makes you a better developer — especially in backend, ETL, reporting, or API design.

• Task

• ValueTask

• async / await

• IEnumerable

• IAsyncEnumerable

• yield return

• yield break

• ThreadPool

• SynchronizationContext

• ConfigureAwait

All floating around, all connected somehow, yet nobody clearly explains how.

Today, we’ll clear the entire sky — not with theory, but with real-life analogies and C# code that make everything stick forever.

Imagine this blog as a guided tour through the C# async universe.

1 — The Core Idea: Synchronous vs Asynchronous

Let’s begin with the root question:

Why do we even need async?

Real-world analogy:

You walk into a restaurant.
The chef cooks one order at a time.
If someone orders biryani (takes 30 minutes), the rest wait.

That’s synchronous execution — one thing at a time.

Async is like:

Chef puts biryani on the stove → while it cooks, he makes dosa → then he makes chai → meanwhile biryani gets ready.

Chef is not idle.

Async != faster.
Async = not blocking the thread while waiting.

This keeps your app responsive and scalable.

2 — What Is a Task?

A Task represents a promise of a result in the future.

It’s like:

“I started making your pizza. I’ll notify you when it’s done.”

Task is a box that will contain the result eventually.

Task<int> CalculateAsync()
{
    return Task.FromResult(42);
}

Task is for async operations that:

• run on another thread

• depend on I/O

• return later

• complete in future

3 — async / await — The Storybook Explanation

async is a keyword for marking a method that can “pause”.

await is a keyword that pauses without blocking the thread.

Real-life analogy:

You order tea.
While tea is boiling, you scroll Instagram.
When the kettle whistles, you continue.

That is await.

public async Task<int> GetNumberAsync()
{
    await Task.Delay(1000); // not blocking
    return 10;
}

The compiler converts async methods into state machines.

4 — ValueTask — When Task Is Too Heavy

Task is a class → reference type → heap allocation.

If the result is often already available, using a full Task object is inefficient.

Example:

public Task<bool> IsOnlineAsync() => Task.FromResult(true);

This allocates a Task for no reason.

ValueTask solves this:

public ValueTask<bool> IsOnlineAsync() => new(true);

Benefits:

• Avoids allocations

• Faster in high-performance code

But:

• More complex

• Should be used in hot paths only

• Avoid returning ValueTask from public libraries (unless needed)

Real-life analogy:

Task → costly UPS shipping box.
ValueTask → envelope.
Use big box only when necessary.

5 — IEnumerable — Synchronous Streaming

IEnumerable<T> lets you return items one-by-one, not the whole list.

public IEnumerable<int> GetNumbers()
{
    yield return 1;
    yield return 2;
    yield return 3;
}

This is synchronous streaming.

Real-life analogy:

You call a friend and ask him to read a book.
He reads line by line, one at a time.
You get each line immediately.

Better for:

• large sets

• streaming data

• pipeline transformations

6 — IAsyncEnumerable — Asynchronous Streaming

But what if each item requires waiting?

Example:
Fetching 1 million records from DB but each chunk is I/O.

IAsyncEnumerable<T> streams items with async pauses.

public async IAsyncEnumerable<int> GetNumbersAsync()
{
    await Task.Delay(100);
    yield return 1;

    await Task.Delay(100);
    yield return 2;
}

Consume it using await foreach:

await foreach (var n in GetNumbersAsync())
{
    Console.WriteLine(n);
}

Real-life analogy:

Friend is reading a book to you but takes a sip of water between lines.

You don’t block.
You wait asynchronously.

7 — yield return and yield break

These keywords create an iterator (state machine).

yield return

Return next item without building a full list.

yield break

Stop the sequence.

public IEnumerable<int> Example()
{
    yield return 1;
    yield return 2;
    yield break;
    yield return 3; // never executed
}

Works for async too, in combination with await.

8 — ThreadPool vs async

Async does not create new threads.

It only frees the current thread during waiting operations.

CPU-bound work should use:

await Task.Run(() => HeavyWork());

I/O-bound work should never use Task.Run.

9 — ConfigureAwait(false)

This is important for libraries.

It tells the runtime:

“I don’t need to resume on original context.”

Useful in:

• libraries

• background services

• console apps

Not recommended in ASP.NET Core (context is already empty).

10 — Common Mistakes & Fixes

Mistake 1: Using .Result or .Wait()

var data = GetDataAsync().Result; // deadlock risk!

Correct:

var data = await GetDataAsync();

Mistake 2: Overusing async/await on simple methods

public async Task<int> Add() => 5 + 10; // unnecessary

Correct:

public int Add() => 5 + 10;

Mistake 3: Mixing I/O and CPU async incorrectly

await Task.Run(() => _db.Save()); // wrong

Correct:

await _db.SaveAsync();

Bringing It All Together — The Final Picture

Here’s the big mental map:

IEnumerable<T>         → sync streaming
IAsyncEnumerable<T>    → async streaming
Task                   → async result in future (heap)
ValueTask              → lightweight async result (stack/struct)
async/await            → pause & resume machine
yield                  → streaming without list
await foreach          → async streaming loop

Everything fits neatly.

Keep the Momentum Going — Support the Journey

If this post helped you level up or added value to your day, feel free to fuel the next one — Buy Me a Coffee powers deeper breakdowns, real-world examples, and crisp technical storytelling.

Final Thought — Async Isn’t Hard. It Was Just Never Explained Simply.

Once you know:

• what Task is (a promise)

• what async/await does (pause & resume)

• when ValueTask matters (performance)

• how IEnumerable streams data

• how IAsyncEnumerable streams asynchronously

…then everything starts making sense.

You stop fearing async.
You start using it properly.
You become a faster, smarter, clearer C# developer.

• scale faster

• share cache across multiple servers

• handle millions of operations per second

• store data in-memory but distributed

• outperform Redis in cluster scenarios

• or build real-time systems like Uber, Netflix, or Shopify

…then Hazelcast is one of the most powerful tools you can learn.

This guide explains Hazelcast in simple language, with a real-world analogy, and gives you step-by-step .NET code examples.

What is Hazelcast (In the Most Human Language Possible)?

Hazelcast is a distributed in-memory data grid (IMDG).

In plain terms:

It’s a superfast, shared in-memory storage system where multiple apps and servers can store and access data together.

Think of it as:

• A cluster of RAM shared across machines

• A very fast distributed cache

• A compute grid for parallel workloads

• A message/event platform

Hazelcast is used for:

• Distributed caching

• Session storage

• Real-time analytics

• Event streaming

• Pub/sub messaging

• Microservice coordination

• Fast key-value storage

It also has built-in:

• CP Subsystems (for strong consistency)

• Cluster discovery

• Automatic failover

• Persistence (if needed)

• SQL engine

And it integrates beautifully with .NET clients.

Real-Time Analogy: Pizza Counters in a Big Restaurant

Imagine you run a massive restaurant with 20 pizza counters.
Customers order pizzas nonstop.

If you stored all orders and workflow in one counter’s notebook:

• Orders become slow

• One counter becomes the bottleneck

• A crash wipes everything

• Other counters have no idea what’s going on

Hazelcast solves this by giving:

• One big shared “in-memory board”

• All pizza counters read & write together

• If one counter crashes, other counters still hold the data

• Notes (data) are replicated everywhere

• Workload is distributed

That’s Hazelcast — a shared memory + shared computing system for your application servers.

Subscribe now

Hazelcast Architecture in Simple Words

Hazelcast consists of:

1️⃣ Cluster (Servers / Members)

• Java server nodes

• Store data in memory

• Replicate partitions

• Perform compute operations

2️⃣ Clients

Your .NET application connects to the cluster as a client using Hazelcast .NET client libraries.

Clients can:

• Get/put data into distributed maps

• Publish/subscribe to topics

• Execute queries

• Run distributed locks

• Use distributed queues

• Execute entry processors

3️⃣ Data Structures

Hazelcast provides:

• IMap (Distributed Dictionary)

• MultiMap

• IList / ISet

• IQueue / ITopic

• CP Subsystem for locks/semaphores

Setting up Hazelcast Server

Step 1 — Download Hazelcast

Download Hazelcast standalone server:

👉 https://hazelcast.com/open-source-projects/download/

Extract and run:

bin/hz-start

You should see logs like:

Members {size:1, ver:1} [
    Member 1: 192.168.1.10:5701 - f5c7a9a2
]

Your Hazelcast cluster (1 member) is ready.

Using Hazelcast in .NET: Step-by-Step

Install the .NET Client

Use NuGet:

dotnet add package Hazelcast.Net

Or via Visual Studio:

Install-Package Hazelcast.Net

Connecting .NET to Hazelcast Cluster

using Hazelcast;

var options = new HazelcastOptions
{
    ClusterName = “dev”
};

options.Networking.Addresses.Add(”127.0.0.1:5701”);

var client = await HazelcastClientFactory.StartNewClientAsync(options);

Console.WriteLine(”✔ Connected to Hazelcast!”);

This creates a .NET client that works like a distributed memory user.

Distributed Map (IMap) — The Most Powerful Structure

Hazelcast’s IMap is basically:

A Dictionary<TKey, TValue> that lives across multiple servers and replicates itself.

Writing data to the map

var map = await client.GetMapAsync<string, string>(”users”);

await map.PutAsync(”user:1”, “Bhargavi”);
await map.PutAsync(”user:2”, “Kishore”);

Console.WriteLine(”Users added!”);

Reading data

var value = await map.GetAsync(”user:1”);
Console.WriteLine(value); // Bhargavi

Updating

await map.PutAsync(”user:1”, “Bhargavi Mojala”);

Deleting

await map.RemoveAsync(”user:2”);

Hazelcast automatically:

• Partitions the map

• Replicates data

• Balances load

• Handles failover

Your .NET code stays simple.

Distributed Locking With Hazelcast

You can build concurrency-safe distributed systems.

var lockObj = await client.GetLockAsync(”payment-lock”);

await lockObj.LockAsync();

try
{
    Console.WriteLine(”Processing payment...”);
    await Task.Delay(2000);
}
finally
{
    await lockObj.UnlockAsync();
}

This ensures only one server in your cluster processes a “payment” at a time.

Ideal for:

• Financial transactions

• Unique ID generation

• Inventory updates

• File processing

Publish/Subscribe Messaging

Hazelcast includes lightweight message broadcasting.

var topic = await client.GetTopicAsync<string>(”alerts”);

await topic.SubscribeAsync(msg =>
{
    Console.WriteLine(”Received alert: “ + msg);
});

await topic.PublishAsync(”High CPU usage!”);

This is simpler than RabbitMQ/Kafka for internal events.

Querying Data Using Predicates

Hazelcast supports distributed querying.

Store complex objects:

public class Product
{
    public int Id { get; set; }
    public string Category { get; set; }
    public double Price { get; set; }
}

Put into map:

var products = await client.GetMapAsync<int, Product>(”products”);

await products.PutAsync(1, new Product { Id = 1, Category = “Electronics”, Price = 25000 });
await products.PutAsync(2, new Product { Id = 2, Category = “Clothing”, Price = 800 });

Query:

var expensiveItems = await products.GetValuesAsync(
    Predicates.GreaterThan(”Price”, 5000)
);

This runs the filter across the cluster and sends minimal data back.

Hazelcast as Distributed Cache for .NET APIs

In ASP.NET Core, register Hazelcast client using DI:

builder.Services.AddSingleton(async _ =>
{
    var options = new HazelcastOptions();
    options.Networking.Addresses.Add(”127.0.0.1:5701”);

    return await HazelcastClientFactory.StartNewClientAsync(options);
});

Use it inside a controller:

[ApiController]
[Route(”api/products”)]
public class ProductsController : ControllerBase
{
    private readonly IHazelcastClient _client;

    public ProductsController(IHazelcastClient client)
    {
        _client = client;
    }

    [HttpGet(”{id}”)]
    public async Task<IActionResult> Get(int id)
    {
        var cache = await _client.GetMapAsync<int, Product>(”products”);

        if (await cache.ContainsKeyAsync(id))
        {
            return Ok(await cache.GetAsync(id));
        }

        // simulate DB call
        var product = new Product { Id = id, Category = “Electronics”, Price = 1200 };

        await cache.PutAsync(id, product);

        return Ok(product);
    }
}

This creates a shared distributed cache across your whole cluster.

Subscribe now

Final Thoughts — Hazelcast Makes .NET Superhuman Fast

Hazelcast is one of those tools that looks complicated from outside but is shockingly simple once you start.

With just a few lines of C#:

• Your app becomes distributed

• Your memory becomes scalable

• Your operations become fail-safe

• Your cache becomes lightning fast

• Your architecture becomes cloud-ready

And best of all —

You don’t need to rewrite your .NET app.
Just plug Hazelcast in.

You now know:

• What Hazelcast is

• How clusters work

• How .NET connects to it

• How to use maps, locks, queues, topics

• How to build distributed systems using simple C#