A Billion Objects, No Index: How Ceph Finds Anything

Object storage sounds simple from the outside.

A client sends an object:

customer-123/profile.jpg

The storage system saves it and returns it later when asked.

Inside a large cluster, the system has to answer a harder question first.

Which physical disk should store this object?

The answer gets harder as the cluster grows:

• Hundreds or thousands of disks
• Many storage servers
• Billions of objects
• Multiple copies of every object
• Different disk sizes
• Servers and disks that fail
• New machines added regularly

The system has to spread objects evenly, keep copies in safe places, and avoid moving data it does not need to move whenever the cluster changes.

Ceph solves this with an algorithm called CRUSH.

CRUSH stands for:

Controlled Replication Under Scalable Hashing

CRUSH calculates where data should live. Instead of keeping a large central table with the location of every object, Ceph works the location out whenever it needs it.

Before CRUSH makes sense, the building blocks of a Ceph cluster have to be clear. This post walks through each one, then puts them together on a single running cluster.

What is Ceph?

Ceph is an open source distributed storage system. It runs on ordinary servers with ordinary disks, and it presents that hardware as storage in three different shapes at the same time.

• Object storage through the RADOS Gateway, usually written RGW. It speaks the S3 and Swift APIs, so applications written for S3 can point at it. This is the path this post follows.
• Block storage through RBD, the RADOS Block Device. It hands virtual disks to virtual machines and containers.
• File storage through CephFS, a POSIX filesystem you can mount like any other.

Underneath all three sits RADOS, short for Reliable Autonomic Distributed Object Store. RADOS is the real storage layer. Whichever interface a client uses, the data ends up as RADOS objects spread over the cluster.

That is why placement is one problem and not three. Solve it once in RADOS and every interface inherits the answer.

The daemons

A Ceph cluster is not a single service. It is a set of processes, each with one job.

Monitors hold the maps. OSDs hold the data. The manager collects metrics and serves the dashboard. The metadata server exists only if you use CephFS, so a cluster that only serves objects runs none.

What makes Ceph different

Most storage systems keep a metadata service that remembers where every piece of data lives. A client asks it for an object and it looks the location up in a table.

Ceph does not do that for data placement. There is no table mapping objects to disks, and no service to ask.

Instead every client and every daemon holds a copy of the cluster map and works the location out for itself. That calculation is CRUSH, and the rest of this post is how it works.

The problem: where should an object be stored?

Imagine a cluster with six disks, numbered Disk 0 to Disk 5.

When a new object arrives, something has to pick a disk. The obvious rule is:

disk = hash(object-name) % number-of-disks

For example:

hash("profile.jpg") % 6 = 2

The object goes on Disk 2. This looks fine until you change anything.

Problem 1: a disk is added

The cluster had six disks:

hash(object-name) % 6

A seventh disk arrives:

hash(object-name) % 7

The divisor changed, so the answer changes for most objects.

An object keeps its home only when its hash gives the same answer under both divisors. That happens for roughly one object in n+1. Everything else relocates.

Relocating is not free. Each moved object costs a read, a network transfer and a write, multiplied by the number of copies you keep.

Problem 2: a disk fails

If Disk 2 dies, the system has to find a new home for everything that lived there.

It also has to know where the other copies of that data are, so it can rebuild from them. Plain modulo says nothing about either.

Problem 3: replicas end up in the same place

Suppose the system keeps three copies of every object. It should not put all three on disks inside one server.

Server A
  Disk 0
  Disk 1
  Disk 2

If all three copies land here and the server loses power, every copy is gone at once.

To avoid that, placement has to understand the physical layout:

Datacentre  ->  Rack  ->  Server  ->  Disk

Problem 4: disks are different sizes

A 16 TB disk should normally hold more than a 4 TB disk.

Plain modulo treats every disk as equal. The small disks fill first, and the cluster reports itself full while most of its capacity sits empty.

CRUSH was designed around all four problems at once.

The basic idea behind CRUSH

CRUSH calculates storage locations from three inputs:

locations = CRUSH(
    placement-id,
    cluster-map,
    placement-rule
)

The output is a list of storage devices:

[osd.1, osd.2, osd.5]

In plain words:

Take the identity of the data, the current shape of the cluster, and the placement policy. Calculate which devices should hold the data.

In Ceph the placement identifier is normally a Placement Group, not the raw object name.

Each part of that chain is explained below, in order.

The building blocks of a Ceph cluster

The next fifteen sections define one term each, in the order CRUSH needs them.

1. Object

An object is the actual piece of data stored in the system.

Examples:

profile.jpg
invoice-2026.pdf
video-123.mp4
customer-record-781
backup-part-0042

An object generally holds three things:

• The object data
• An object name or identifier
• Optional metadata

Objects are not arranged in folders on one disk the way files are. The object name is an input to the placement calculation, not a path to a location.

2. Node

A node is a physical or virtual server in the cluster, such as Node A, Node B, Node C.

Each node usually holds several storage devices.

A node is the machine. A disk is a storage device inside that machine.

The distinction matters because CRUSH can spread replicas across different nodes:

Replica 1  ->  Node A
Replica 2  ->  Node B
Replica 3  ->  Node C

If Node A fails, the other two copies stay available.

3. OSD

An OSD is the Ceph component that stores object data.

OSD stands for:

Object Storage Daemon

An OSD is normally tied to one storage device.

The OSD is not the disk itself. It is the Ceph service responsible for managing data on that disk.

An OSD does work such as:

• Storing objects
• Reading objects
• Replicating data to other OSDs
• Recovering missing data
• Checking data integrity by scrubbing
• Communicating with other OSDs

A useful mental model:

Node = the server
Disk = the physical storage device
OSD  = the Ceph service managing that device

A node can run many OSDs, because it can hold many disks. A server with twelve disks runs twelve OSDs.

This is why placement cannot work at the disk level alone. Several OSDs sound like several separate places. If they share a node they also share a power supply, a motherboard and a network card.

4. Pool

A pool is a logical collection of objects that share the same storage policy.

A company might create separate pools for images, videos, backups and database-volumes.

A pool is not a server, a disk or a folder. It is a logical storage boundary.

It is also the container for placement groups. Every pool is divided into a fixed number of them, and every object you write to the pool lands in exactly one.

That is what makes the pool the unit that matters. It is where the policy is set, and it is the boundary inside which objects get bucketed for placement. Section 6 covers what a placement group actually is.

Each pool carries its own configuration:

images pool
  Use SSD devices
  Keep 3 copies
  Spread copies across different servers
  Use 128 placement groups

Another pool can use a completely different policy:

backups pool
  Use HDD devices
  Use erasure coding
  Spread fragments across different racks
  Use 256 placement groups

This is how one Ceph cluster supports different kinds of workload at the same time.

5. Pool ID

Every pool has a readable name and an internal numeric ID.

Pool name: images     Pool ID: 7
Pool name: videos     Pool ID: 8

Ceph uses the numeric pool ID internally.

The ID matters because two pools can hold objects with the same name:

Pool 7 / logo.png
Pool 8 / logo.png

Those are different objects, because they belong to different pools.

A pool name can also be renamed later. The pool ID stays as the stable identity.

The pool ID is not a host number, a disk number or an OSD number. It identifies the logical pool and nothing else.

6. Placement Group

A Placement Group, usually written PG, is a logical group of objects.

Suppose a pool holds one million objects. Ceph does not want to track placement, recovery and health separately for all one million. So it divides them into a much smaller number of groups.

1,000,000 objects  ->  128 placement groups

Every object belongs to exactly one placement group.

PG 0    object-A, object-F, object-X
PG 1    object-B, object-G, object-M
PG 2    object-C, object-D, object-Z

A placement group is not:

• A disk
• A host
• An OSD
• A physical partition
• A group of servers

It is a logical grouping Ceph uses to manage many objects together.

Objects in the same PG normally use the same OSD set:

PG 7.1d  ->  [osd.1, osd.2, osd.5]

Every object that maps to PG 7.1d is stored on those OSDs. The objects stay separate. They share a placement decision and a recovery decision, nothing else.

7. Number of placement groups

Every pool has a configured number of placement groups, set by:

pg_num

For example:

Pool:   images
pg_num: 128

Objects in the images pool are spread over 128 placement groups, numbered 0 to 127.

When an object is written, Ceph hashes the object identifier and maps the result into one of the pool's PGs.

Teaching material usually writes this as:

PG number = object hash % pg_num

The real calculation masks with pg_num - 1 when pg_num is a power of two, and uses a stable variant when it is not. The idea is unchanged:

Hash the object name, and the hash selects one PG inside the pool.

Why not one PG per object?

Because PGs are also the unit for:

• Placement
• Peering
• Recovery
• Backfilling
• Scrubbing
• Health tracking

One PG per object would create enormous memory, CPU and coordination overhead.

Why not a single PG?

If a large pool had only one PG, every object in it would use the same small set of OSDs. Most disks in the cluster would sit unused for that pool.

PGs let a pool's objects spread over many different OSD combinations.

Too few, too many

Too few PGs causes uneven distribution, some OSDs holding far more than others, and poor use of the cluster.

Too many PGs causes higher memory use, more peering work and more cluster management overhead.

Current Ceph ships a PG autoscaler that picks and adjusts this number for you.

8. PG ID

A PG number only means something inside a particular pool. So Ceph combines the pool ID and the PG number.

Pool ID:  7
PG value: 5

is displayed as:

7.5

That is not the decimal number seven point five. It reads as pool 7, placement group 5. The dot separates the two parts.

Another PG might be:

7.1d

which means pool 7, PG value 1d. Ceph prints the PG part in hexadecimal, so 1d is decimal 29.

A PG ID is not a host address or an OSD address. It identifies one logical placement group inside one pool.

9. Cluster map

Ceph components need a shared view of the cluster. That view lives in a set of maps.

A cluster map holds information such as:

• Which OSDs exist
• Which OSDs are reachable
• Which OSDs are part of the cluster
• Pool configuration
• Replication settings
• OSD weights
• Cluster topology
• CRUSH rules

The short version:

The cluster map is the current description of the storage cluster.

Ceph keeps several specialised maps internally. Two matter here.

The OSD map

The OSD map holds OSD state and pool configuration.

osd.0 is up and in
osd.1 is up and in
osd.2 is down

up means the daemon is running. in means it should be holding data.

The CRUSH map describes the storage hierarchy and the weight of every device. It is the one that decides placement, so it gets its own section next.

When people write cluster-map in a simplified CRUSH formula, they mean all of the placement-related cluster information Ceph has.

10. CRUSH map

The CRUSH map describes how storage is physically organised.

The hierarchy can include many levels:

Root -> Datacentre -> Room -> Row -> Rack -> Host -> OSD

Not every cluster uses every level. A smaller cluster may declare only:

Root -> Host -> OSD

Ceph does not mind which levels you use. It cares that you declared them, because those levels are exactly what it can spread copies across.

11. Failure domain

A failure domain is a part of the infrastructure that can fail as one unit.

Examples:

• OSD
• Server
• Chassis
• Rack
• Room
• Datacentre

Suppose a pool keeps three copies of every object. If the failure domain is host, CRUSH puts the copies on different hosts.

The unsafe placement above puts two copies on one node. Losing that node removes two of the three copies at once.

For a larger cluster the failure domain may be rack:

Replica 1  ->  Rack A
Replica 2  ->  Rack B
Replica 3  ->  Rack C

That protects the data even when a whole rack loses power or networking.

12. CRUSH rule

The CRUSH map describes what infrastructure exists. The CRUSH rule describes how data should be placed on it.

A rule says, in effect:

Start from the default storage root.
Use only SSD devices.
Choose three different hosts.
Choose one OSD from each host.
Return those OSDs.

A replicated rule is written like this:

rule replicated_ssd {
    take default class ssd
    chooseleaf firstn 0 type host
    emit
}

The exact syntax matters less than the meaning of each line.

take

This picks where in the hierarchy selection begins.

take default

means start from the root named default.

Device class

A rule can restrict placement to one device type, such as SSD only, HDD only or NVMe only. Ceph detects device classes automatically, so a rule can target fast media without listing individual disks.

chooseleaf

This means:

Choose a failure-domain bucket, such as a host, then choose an OSD beneath it.

If the failure domain is host, Ceph picks distinct hosts first, then one OSD inside each. Change type host to type rack and the same rule spreads across racks instead.

emit

This returns the selected OSDs.

A rule is a short set of placement instructions, nothing more. Newer Ceph (Squid) also adds MSR rules, which allow multiple OSDs inside one failure domain by retrying earlier steps when an out OSD is hit.

13. Replication size

A replicated pool has a setting called:

size

For example:

size = 3

That means Ceph maintains three copies:

1 primary copy
2 replica copies

Placed, for example, as:

osd.1  ->  primary
osd.2  ->  replica
osd.5  ->  replica

The CRUSH rule decides the placement restrictions. size decides how many OSDs the rule has to produce. size = 2 needs two, size = 4 needs four.

There is a second setting that people often miss:

min_size = 2

min_size is the fewest copies the pool will still accept a write on. With size = 3 and min_size = 2, the pool keeps taking writes after one OSD fails and refuses them if a second one goes. That setting is what stops a degraded pool from quietly running on a single copy.

The cluster must also have enough distinct failure domains to satisfy the rule. A rule asking for three different hosts cannot be satisfied on a two-host cluster.

14. OSD weights

Not every OSD has the same capacity. A cluster bought in waves ends up with mixed disk sizes:

osd.0  ->  4 TB
osd.2  ->  8 TB
osd.6  ->  16 TB

CRUSH represents relative capacity with a weight. By default the weight is the capacity in TiB, so a 4 TB device gets about 3.64 and an 8 TB device about 7.28. The numbers below use 4, 8 and 16 to stay readable.

Over many PGs, the larger device is selected more often. This does not mean any single placement follows an exact ratio. It means the distribution approaches the weighted ratio across a large number of PGs.

Weight is also a control, not only a description. To retire a disk, lower its weight toward zero and CRUSH moves its PGs onto other devices with no downtime.

Two details matter here. The new target placement applies immediately; what takes time is the data movement, which runs asynchronously through recovery and backfill and is rate limited. And there are two separate knobs:

• CRUSH weight, set with ceph osd crush reweight, is normally the device's relative capacity.
• The reweight override, set with ceph osd reweight, is a temporary 0 to 1 multiplier on top of it.

15. Monitor

Ceph Monitors maintain the authoritative cluster state. They track:

• Which OSDs are available
• Cluster map versions
• Pool configuration
• Membership and health information

A client authenticates against a monitor and receives the current cluster maps. It keeps receiving newer map versions as the cluster changes:

Client  ->  request cluster map  ->  Monitor
Client  <-  current map          <-  Monitor

After that the client calculates the target OSDs itself. The monitor does not handle every object read and write.

This is what makes the design scale. The data path is:

Client  ->  Primary OSD  ->  Replica OSDs

rather than:

Client  ->  Central metadata server  ->  Storage server

How the complete placement process works

Now put the pieces together on one cluster.

The cluster is the one from the CRUSH map above: two racks, four nodes, eight OSDs.

Node A    osd.0, osd.1
Node B    osd.2, osd.3
Node C    osd.4, osd.5
Node D    osd.6, osd.7

The pool configuration is:

Pool name:        images
Pool ID:          7
Number of PGs:    128
Replication size: 3
Minimum size:     2
Failure domain:   host
Device class:     SSD

The client wants to store customer-123/profile.jpg.

Step 1: select the pool

The application writes the object to the images pool, which is pool ID 7.

The pool determines which storage policy applies.

Step 2: hash the object identifier

Ceph hashes the object identifier:

hash("customer-123/profile.jpg")

Suppose the illustrative result is 29341. The actual hash value and mapping are internal to Ceph.

Step 3: map the object to a PG

The images pool has 128 PGs, and Ceph maps the hash into one of them.

29341 mapped into 128 PGs  ->  PG 29

The object belongs to PG number 29 inside pool 7. Decimal 29 is hexadecimal 1d, so the PG is displayed as:

7.1d

Step 4: apply the CRUSH rule

The pool's rule says: use SSD devices, keep three copies, place them on different hosts.

CRUSH receives four things:

Placement input:   PG 7.1d
Cluster topology:  nodes, OSDs and weights
Placement rule:    three SSD OSDs on different hosts
Copies required:   3, taken from the pool's size

CRUSH calculates:

[osd.1, osd.2, osd.5]

Their physical locations are Node A, Node B and Node C. The result satisfies the rule because every selected OSD sits on a different host.

Step 5: select the primary

The selected OSDs form an ordered set. The primary for the PG is the first OSD of its acting set, which normally matches this CRUSH result but can differ during recovery.

Primary:   osd.1
Replica 1: osd.2
Replica 2: osd.5

The primary is not a permanent leader for the whole cluster. It is the primary for this one PG. A different PG has a different primary:

PG 7.1d  ->  [osd.1, osd.2, osd.5]   primary osd.1
PG 7.34  ->  [osd.4, osd.0, osd.3]   primary osd.4

Across 128 PGs, the primary work spreads over every OSD in the pool.

Step 6: send the write

The client sends the object to the primary OSD and to nothing else.

One thing to be clear about here. "The client" means the RADOS client, which is not usually your application. An application talking S3 sends its request to RGW, and RGW is the thing that holds the cluster map, runs CRUSH and talks to OSDs:

Application      ->  S3 request to RGW
RGW (librados)   ->  the RADOS client: computes the PG and the primary OSD
Primary OSD      ->  coordinates the replicas

RBD works the same way through librbd or the kernel client, and CephFS through its own client. Your code does not download the CRUSH map.

The primary coordinates its own write and the replica sub-operations together, rather than finishing locally and only then forwarding. It acknowledges the client once the required OSDs have completed the operation.

Every object that hashes into PG 7.1d follows this exact path to these exact three OSDs.

Why not map every object directly with CRUSH?

At first the placement group layer looks unnecessary. Why not calculate Object -> OSDs instead of Object -> PG -> OSDs?

Because PGs give Ceph a manageable unit for cluster operations.

Suppose a million objects need recovery after an OSD failure. Handling each object independently would create enormous coordination overhead. Instead Ceph reasons about the affected PGs:

PG 7.1d needs recovery
PG 7.20 needs recovery
PG 7.31 needs recovery

Each of those PGs may hold many thousands of objects.

PGs are therefore the unit for:

• Deciding placement
• Tracking which OSDs own data
• Coordinating replicas
• Detecting inconsistent copies
• Recovering after failures
• Moving data during rebalancing
• Scrubbing and integrity checks

A PG is the middle layer between individual objects and physical OSDs.

What happens when the cluster changes

The cluster map is never static. Disks die. Servers die. New hardware arrives. Each of those changes the input to CRUSH, and each moves a different amount of data.

An OSD fails

Suppose PG 7.1d lives on [osd.1, osd.2, osd.5] and osd.2 fails.

The monitors learn the OSD is unavailable and mark it down in a new cluster map. down only means the daemon is unreachable. Nothing is replaced yet.

If it stays unavailable, Ceph marks it out after mon_osd_down_out_interval, which defaults to 10 minutes. out is the signal to stop placing data there, and that is what starts the remapping below.

Once the OSD is out, CRUSH calculates a replacement placement using the new map. Another valid OSD exists on a host not already used by this PG:

osd.7 on Node D

So the new placement becomes:

[osd.1, osd.7, osd.5]

Ceph then recreates the missing replica on osd.7, copying from osd.1 or osd.5.

While this runs, the PG is marked degraded because it holds fewer copies than the pool requires. Clients keep reading and writing throughout, because min_size is 2 and two copies are present.

After recovery finishes, the required redundancy is restored.

A complete node fails

Suppose Node B holds osd.2 and osd.3. If the node loses power, both OSDs become unavailable at the same time.

Because the placement rule spread replicas across hosts, the other copies are on different nodes:

Replica 1  ->  Node A: available
Replica 2  ->  Node B: unavailable
Replica 3  ->  Node C: available

No PG held more than one copy on Node B, so no PG lost more than one copy. Ceph rebuilds the missing copies on the remaining nodes.

Had the rule said type osd instead of type host, some PG would have ended up with two copies on Node B. That PG would now be down to one copy and one failure away from gone.

This is why topology-aware placement beats picking disks without knowing which server holds them.

A new OSD is added

Suppose a new node arrives with more OSDs. The CRUSH map now contains additional storage devices.

For some PGs the calculated placement changes:

Before:  PG 7.1d  ->  [osd.1, osd.2, osd.5]
After:   PG 7.1d  ->  [osd.1, osd.8, osd.5]

Ceph moves the necessary data onto the new OSD.

The important part is that CRUSH avoids remapping everything. Only a portion of the data moves, enough for the new device to take its fair share. That is far better than a modulo scheme, where changing the disk count can relocate most objects.

The reason a map change moves so little is the bucket algorithm underneath, called straw2. Every candidate device draws a weighted value and the highest draw wins the slot. Adding or removing one candidate only changes the comparisons that candidate took part in, so most PGs keep the winners they already had.

An OSD is removed

Removing an OSD also changes the cluster map. PGs that used it must be reassigned.

Before:  PG 7.1d  ->  [osd.1, osd.2, osd.5]
Remove osd.2
After:   PG 7.1d  ->  [osd.1, osd.4, osd.5]

Ceph copies or reconstructs the required data on the replacement OSD.

PGs that never used osd.2 do not move at all. That is what limits unnecessary movement.

Telling the pieces apart

Three pairs of terms get mixed up more than any others. Each pair is worth stating side by side.

Pool and PG

A pool is the logical storage container and the policy boundary. A PG is one logical placement group inside that pool.

Pool: images
  PG 7.0
  PG 7.1
  PG 7.2
  PG 7.3
  ...

The pool defines settings such as:

• Replication or erasure coding
• Number of placement groups
• CRUSH rule
• Device class
• Minimum replica requirements

The PG groups objects and maps them to an OSD set.

A short summary:

Pool  ->  defines the storage policy
PG    ->  groups objects for placement and recovery
OSD   ->  stores the actual data

Node and OSD

A node is a server. An OSD is a Ceph storage service normally managing one disk inside that server.

Node A
  osd.0
  osd.1

If Node A fails, both OSDs fail together.

That is why a CRUSH rule should usually select replicas across different nodes rather than just selecting different OSD IDs. Picking:

osd.0, osd.1

looks like two devices, but it is unsafe if both belong to Node A.

CRUSH map and CRUSH rule

The CRUSH map describes the cluster. It answers:

Which racks exist?
Which nodes belong to each rack?
Which OSDs belong to each node?
What is each OSD's weight?
Which devices are SSDs or HDDs?

The CRUSH rule describes the placement policy. It says:

Use SSDs.
Choose three different hosts.
Choose one OSD from each host.

A simple way to hold the difference:

CRUSH map  = the map of the building
CRUSH rule = the instructions for choosing rooms

The map says what exists. The rule says how to choose from it.

What CRUSH returns

CRUSH returns an ordered list of OSDs:

[osd.1, osd.2, osd.5]

That list is the intended placement for the PG under the current map and rule. The primary is the first OSD of the acting set, which is normally this same list:

osd.1  ->  primary
osd.2  ->  replica
osd.5  ->  replica

The primary coordinates client operations for every object in that PG.

Up set and acting set

Two terms show up in Ceph output and they are easy to confuse.

The up set is the placement calculated from the current CRUSH and OSD maps. It is the answer.

The acting set is the group of OSDs currently responsible for serving the PG. It is the reality.

Most of the time they are the same:

Up set:     [osd.1, osd.2, osd.5]
Acting set: [osd.1, osd.2, osd.5]

During recovery, migration or temporary remapping, they differ.

CRUSH can decide that osd.7 should hold a copy long before osd.7 actually has one. If the acting set switched over instantly, clients would be sent to an OSD holding nothing. So the acting set lags on purpose and catches up once the data lands.

There is a second override worth knowing, and it works the other way around. pg_temp changes the acting set, as above. The manager's balancer instead records pg-upmap entries that change the up set, which is CRUSH's own answer. Its default mode is upmap.

So the final placement in a real cluster is the CRUSH result plus any upmap exceptions laid on top, not the raw CRUSH result alone.

The short version:

CRUSH calculates where the PG should be. The acting set is the OSDs serving it right now.

Replicated pools and erasure-coded pools

CRUSH supports more than simple replication.

Replicated pool

With size = 3, each selected OSD stores a complete copy of the object.

Object
  Full copy on osd.1
  Full copy on osd.2
  Full copy on osd.5

Erasure-coded pool

With a configuration such as 4+2, the object is split into four data fragments and two parity fragments. CRUSH is asked for six OSDs instead of three, and the rule spreads those fragments across different hosts or racks.

Both layouts keep the data after two OSD losses, and the erasure-coded one uses half the disk space. Neither keeps serving I/O there. A size = 3 pool is down to one copy, below its min_size of 2. A 4+2 pool is down to four shards, below the recommended min_size of k+1, which is 5. The data is recoverable in both cases, but writes block until redundancy comes back.

The trade is CPU and latency. Reading a whole object gathers all k data shards from as many OSDs, though a small aligned read may touch fewer, and a degraded read has to pull enough surviving shards to reconstruct what is missing. Partial writes are more expensive too. Replication is faster and erasure coding is cheaper. A real cluster runs both, in different pools. Our object storage system design walks through a 10+4 layout at exabyte scale for exactly this reason.

One practical limit is worth stating. A 4+2 rule with failure domain host needs at least six hosts. On the four-node cluster used throughout this post, that rule could not be satisfied at host level.

The placement idea does not change:

Object  ->  Placement Group  ->  CRUSH rule  ->  OSD locations

The only difference is whether each chosen OSD stores a complete replica or an erasure-coded fragment.

Why CRUSH scales

A traditional system might keep a central table:

Object A  ->  osd.1, osd.2, osd.5
Object B  ->  osd.4, osd.8, osd.9
Object C  ->  osd.3, osd.6, osd.10

For billions of objects that table gets very large. It also becomes a bottleneck, because clients have to keep asking where objects live.

CRUSH keeps no central location record for any object. Any component holding the same map performs the same calculation:

CRUSH(PG, map, rule)

The result is deterministic:

Client: CRUSH(PG 7.1d, map version 100, rule X)  ->  [osd.1, osd.2, osd.5]
OSD:    CRUSH(PG 7.1d, map version 100, rule X)  ->  [osd.1, osd.2, osd.5]

Both get the same answer because both used the same inputs. That is what lets a client talk directly to the right storage OSD, with no lookup service in between.

CRUSH is deterministic, not permanently fixed

Deterministic means:

Same placement input + same map + same rule = same result

The result can still change when one of those inputs changes.

A different PG

PG 7.1d  ->  [osd.1, osd.2, osd.5]
PG 7.2a  ->  [osd.4, osd.0, osd.3]

That is the point of having 128 of them.

A different cluster map

An OSD fails or a new OSD is added:

Old map  ->  [osd.1, osd.2, osd.5]
New map  ->  [osd.1, osd.7, osd.5]

A different rule

Changing host-level placement to rack-level placement produces a different OSD set:

Host-level rule  ->  one OSD from each host
Rack-level rule  ->  one OSD from each rack

The calculation is repeatable. Its output reflects the current cluster and the current policy.

CRUSH compared with ordinary consistent hashing

Consistent hashing and CRUSH solve related problems, but CRUSH understands storage topology.

A basic consistent-hashing system answers:

Which server owns this key?

CRUSH answers a more detailed question:

Which weighted storage devices should hold all copies or fragments of this data, while respecting host, rack or datacentre failure boundaries?

CRUSH supports:

• Weighted devices
• Multiple replicas
• Host-aware placement
• Rack-aware placement
• Datacentre-aware placement
• Device classes
• Erasure-coded fragments
• Controlled data movement when the cluster changes

The simplified comparison:

Consistent hashing:  key -> node
CRUSH:               PG + topology + policy -> ordered OSD set

Common misunderstandings

Is a PG a host?

No. A PG is a logical group of objects. A host is a physical or virtual server.

PG    ->  logical placement unit
Host  ->  server

Is a PG an OSD?

No. A PG is mapped to an OSD set.

PG 7.1d  ->  [osd.1, osd.2, osd.5]

Does a pool live on one node?

No. A pool is distributed across many OSDs and nodes, usually most of the cluster.

Is the pool ID a server ID?

No. The pool ID identifies a logical pool and nothing physical.

Does PG ID 7.5 mean seven point five?

No. It means pool 7, placement group 5. The PG part is printed in hexadecimal.

Does the client write separately to every replica?

No. The client sends the operation to the primary OSD, and the primary coordinates the other replicas.

Does the monitor store every object location?

No. The monitor maintains cluster maps and state. Object placement is calculated from the PG, the maps and the CRUSH rules.

Does every object in a PG become one large object?

No. The objects stay separate. They share placement and recovery management, nothing more.

A simple mental model

Think of Ceph as a large warehouse system.

Pool. A storage category with its own rules, such as Electronics, Furniture or Documents.

Object. An individual item, such as a laptop box, a chair package or a document envelope.

Placement group. A numbered logical group used to organise many items for placement.

Node. A warehouse building.

OSD. A storage section inside the building that actually holds items.

CRUSH map. The complete map of buildings, rooms and storage sections.

CRUSH rule. The safety instructions:

Keep three copies.
Use three different buildings.
Use climate-controlled sections only.

CRUSH. Takes the item's placement group, the warehouse map and the safety instructions, then calculates the storage sections.

The analogy is not exact, but it holds the relationships between the main pieces.

Complete terminology summary

The complete process in plain English

When an application stores an object in Ceph:

1. The application chooses a pool.
2. Ceph hashes the object identifier.
3. The object is assigned to one placement group inside that pool.
4. Ceph takes the PG identifier and the pool's CRUSH rule.
5. CRUSH examines the cluster hierarchy, device classes and weights.
6. CRUSH selects the required number of OSDs while respecting failure-domain rules.
7. The client sends the write to the primary OSD.
8. The primary coordinates the other replicas or erasure-coded fragments.
9. If an OSD fails, Ceph recalculates the affected placement and restores the missing data elsewhere.
10. If a new OSD is added, Ceph moves only the data needed to give it a fair share.

The complete flow:

Object
  ->  Pool
  ->  Hash
  ->  Placement Group
  ->  CRUSH map + CRUSH rule
  ->  Primary and replica OSDs
  ->  Physical storage devices

Final takeaway

The main problem in a distributed object-storage system is not saving an object. The real problem is deciding:

• Where the object should be stored
• Where its replicas should be placed
• How to avoid correlated failures
• How to distribute data according to device capacity
• How to recover after failures
• How to add or remove storage without moving everything

Ceph solves this with two layers.

First, objects are grouped into placement groups:

Object  ->  PG

Then CRUSH maps each PG to physical storage devices:

PG + cluster topology + placement rule  ->  OSD set

In one sentence:

CRUSH is a deterministic, weighted and topology-aware algorithm that calculates where Ceph should store data, without a central per-object location table.

The relationships worth remembering:

Pool             ->  defines the storage policy
Placement Group  ->  groups objects for placement and recovery
CRUSH map        ->  describes the storage topology
CRUSH rule       ->  defines placement constraints
OSD              ->  stores the actual object data
Node             ->  hosts one or more OSDs

This post covered how Ceph decides where data goes. It did not cover whether you should run Ceph at all. That is a separate question, and for a small team the honest answer is often no.

Our Ceph technology page works through the other side. It covers where Ceph is worth its operational cost, and where MinIO or a managed service is the better call. It also covers the failure modes that catch real clusters, such as recovery storms that saturate the network.