Once you have more than one AI machine, ordinary gigabit networking becomes the bottleneck. Copying a 100 GB model takes about 15 minutes at 1 Gb/s and under 10 seconds at 100 Gb/s. Here is how to build a fast network for a small lab.
Step 1: decide what needs the speed
- Two GB10 systems only: no switch needed. Link them directly with a QSFP112 cable (see our clustering guide).
- A few workstations sharing models and datasets from a NAS: 25G to each machine is plenty, with 100G between the switch and the storage.
- Three or more nodes training or serving together: look at 100G or faster per node, and at InfiniBand switches for larger clusters.
Step 2: pick a switch
| Switch | Ports | Good for |
|---|---|---|
| MikroTik CRS504-4XQ-IN | 4 × 100G (each splits to 4 × 25G) | Desk-side lab, 2–4 fast machines |
| MikroTik CRS510-8XS-2XQ-IN | 8 × 25G + 2 × 100G | Small team, compact |
| MikroTik CRS518-16XS-2XQ-RM | 16 × 25G + 2 × 100G | Rack, many 25G machines |
For larger GPU clusters, see the NVIDIA Quantum and Spectrum switches in our switching collection.
Step 3: cables, DAC or optics?
- DAC (direct-attach copper): cheapest, no power, very reliable, but only reaches a few metres. Use it inside a rack or across a desk.
- AOC (active optical cable): a fibre cable with the ends built in, light and thin, and reaches tens of metres.
- Transceivers + fibre: the most flexible option for longer runs, or when cabling runs through walls or between rooms.
Match the form factor to the port: SFP28 for 25G, QSFP28 for 100G, and QSFP112/OSFP for 200G and above. Browse cables and optics.
Step 4: breakout cables
A breakout cable turns one 100G QSFP28 port into four 25G SFP28 connections, so the 4-port CRS504 can serve up to 16 machines at 25G. On MikroTik switches, set each of the four sub-ports to a fixed 25G speed (and turn off auto-negotiation if the link doesn't come up on its own) in RouterOS. Each sub-port then appears as its own interface, for example qsfp28-1-1 to qsfp28-1-4.
Step 5: basic setup
- Update RouterOS to the latest stable version first.
- Put the ports your machines use in one bridge so they're on the same network, with hardware offloading enabled.
- Turn on jumbo frames (MTU 9000) on the switch and on every machine if your workloads move big files. It must match end to end.
- Test with
iperf3between two machines before relying on it.
A typical small-lab layout
Two or three GPU workstations at 25G each via breakout, a NAS at 100G, and one uplink to your office network: all from a single CRS504 on a shelf. Want it as one order? Our Small AI Lab 100G Network Kit has the switch and cables for exactly this, and the Small Team 25G Network Kit covers four workstations plus a NAS. Want us to check a design? Get in touch.
