ASRock 1U12LW-C2750

Because there are so few reviews to find, I will continue in English to reach a global audience.

Shipping

Carton box

It's handy that the identification label is duplicated. Both labels are attached on opposite black sides of the outer carton box. The also is an inner carton box, white soft foam, a transparent plastic bag, and finally in there is the barebone server.
[img]Eq9ZEvG0[/img]

Box dimensions

The outer box I do measure at 106 x 63 x 23. It carries a gross weight of 21,39 kilograms according to the label, the nett weight should be 11,70 kg. And it says that the thingy is made in "Taiwan". I haven't checked it.

Unboxing

When opening both outer and inner cartons boxes first reveals the white soft foam that carries a long carton box containing the rack rails.
[img]gmQJF4S6[/img]
[img]Eq9ZEvG0[/img]

What inside the box

Spare parts

The smaller carton box carries a spare fan, a plastic bag with 8 screws to mount 2.5" hdd's, a drive cd-rom, 6 pcs of 130mm cable ties in a grip bag, a plastic bag with 48 screws #6-32xL4x0.8 to mount 12 3.5" h.d.d., 3 fan cover sponges in a plastic bag, a 1U12LIW_RB1 PCI express riser card with open slot (so you can insert cards that are longer than x8), and a quick installation guide. Although the system comes pre-assembled.
[img]JG1YKGPm[/img]
[img]0G782Chd[/img]

Fans

The spare fan is of identical make and model as the 5 fans that are pre-assembled. The DC fan make is "NMB" and the model is "04028DA-12R-AUF". Its a 12V DC 0.52A fan that is made in China and according to the datasheet runs between 0 and 18000 rpm. In the latter case the fan moves 20.1 CFM at 55 dB of noise and a power consumption of 5.16 watts. With Asrock firmware 2.90, the lowest configurable fan speed is 30% PWM duty cycle, which equals to 7900/8000 rpms. The spare fan also has its 2 vibration decouplers already mounted (that saves assembly time).
[img]hArGlUEc[/img]

Rails

The rack rails are of a tool-less type. According to the packaged 3-page instructions leaflet the maximum carrying weight is 100 lb (45,4 kg). Also according to this leaflet is that it doesn't work with 1/2" pitch, as it is build for 5/8" pitch EIA bracket, both round/square holes. Minimum required distance between the rack posts is 727 millimeters and the maximum inner post distance is 992 millimeters.
[img]RL30X6En[/img]
The sliding rails have the marking "TAIWAN 3514" and "PAT:M473038" engraved, and on each side (front and rear) a large letter "L" or "R" for left versus right side mounting, the head itselfs also contains a much smaller embossed "L" or "R". There are two sliding sections, each with their own locking mechanism. The first step expands to 44 centimeters, and locks, the second step expands another 42 centimeters and locks. The inner rail needs to be mounted to the chassis with M4 screws. There are 12 pcs of M4x6 included.
[img]nPpiOaEP[/img]
[img]wKIHgv1L[/img]
[img]OmTbNQu3[/img]
[img]E0ksonpz[/img]

More unpacking

Futher unpacking reveals the chassis. The metal sheets are still covered in a blue semi-transparent protective cover. Attached is a bag of silica gel to absorb moisture.
[img]MZslgUhx[/img]

Chassis

The chassis cover is in 2 parts. The larger front part covers all 12 3.5" HDD positions, where the smaller rear part covers the mainboard, PSU and fan positions. Handy is that these covers are tool-less screws. Also the 12 pieces of 3.5" hard disk drive carriers are tool-less.
[img]29TfXEOL[/img]
[img]1N5ZEDyn[/img]
[img]8xvEEhon[/img]

PSU: ENP-7025B

After removing the rear chassis cover the Enhance ENP-7025B PSU becomes visible. The 150 mm short 81.5 mm wide PSU carries a product identification label as well as an 80 plus bronze label. According to its product label the PSU is rated for 100-240V AC at 60-50Hz.

Rated DC rail output
+3.3V	+5V	+12V1	+12V2	-12V	5V standby
13A	14A	18A	18A	0.3A	2A

The PSU contains a brushless DC fan. According to the 80plus ENP-7025 test report the PSU runs at an efficiency level of 83.55% at 20% load (59.94 watts in and 50.08 watts out). That was measured at the US standard of 115VAC 60Hz. Inside is the PSU is an ADDA AD0412HB-C50 brushless 40x40x20 DC fan. The fan specs are 12V, max. 0.11A, 1.32W. Adda datasheet lists 7800 RPM at 8.80 CFM.
[img]AwsCPoCe[/img]

Cabling

Power cables are nicely routed to the front through an opening that has white plastic covering around the sharp metal edges.
[img]2RHEOKvN[/img]
Also the data cabling has neat labels near each SATA 7-pin data cable connector.
[img]zkWinfTN[/img]
Also the 2 pcs of 2.5" drive positions are neatly routed with SATA 15-pin power cables.
[img]xaMnb6Jj[/img]
But why route these power cables in front of the fan airflow (they don't need that much cooling).
[img]jW0Mp8Te[/img]
Near the CPU and memory slots this power cabling mess is nearly obstructing the complete airflow: 0 points to Asrock for this.
[img]fc1FITqI[/img]
Neither are there any points awarded for routing the data cables through sharp edged metal sheet holes without any protective covering. Are data cables that much less important then the power cables?

Lithium battery

For the moments when there is no power supplied, the real time clock and CMOS values are kept on track via battery energy. For this purpose the C2750D4I mainboard holds a 3 volts lithium CR2032 coin cell, produced by Newsun. The voltage I measured is was only 3.03 volts in circuit, and 3.08 out of circuit. The BMC "BAT" reading is 2.98 volts.
[img]VMdBCnwn[/img]

When to replace the lithium battery?

The Asrock C2750D4I motherboard manual at page 56 and 57 isn't very clear at which level the battery should be replaced: "~3VDC. Install a new battery if it does not.". Does this mean that I should replace or can it wait till voltage is dropped to 2.50 or 2.75 volts (significant numbers)? Battery manufacturers do use 2.0 volts as CR2032 cut-off voltage in their datasheets. The CR2032 battery brand new from the factory is specified between 3.0 and 3.4 volts. For specifying 3 volts in the manual, resulting in unnecessary polluting the environment, I do award negative points to Asrock here (and that is not the last time the Asrock manual gets negative points).

Port labels are missing

The machine is going to be plugged. Power is obvious with only one IEC C14 inlet. VGA is obvious too. But which port is for the BMC, and which one is LAN1/LAN2? Again, zero points for Asrock here, due to the unnecessary lookup in the manual here.
[img]6jEkRzzQ[/img]
Left and above the USB connectors is the BMC RJ45 connector and LAN1 is the upper and LAN2 the lower RJ45 socket.

Stock firmware

I did some more examination of the motherboard. Factory stock C2750D4I comes with motherboard firmware P2.80 according to the white label sticked on the PCB component. When opening the UEFI firmware I do read P2.60W however. The BMC comes labelled with firmware version 0.23.0. In the Dashboard the BMC Firmware Revision is listed as 4.2.0 (Nov 19 2014).
I flashed the BMC firmware with the most recent downloadable version, that is 0.27.0 (Jan 13 2016).

P2.60W DDR3 compatible modules

The first DIMM slot to install is DDR3_A1 according to the manual. That is the blue slot between the PSU and CPU. To extend the memory compatibility list, I can say that there is not any issue at all with any DDR3 memory module I plugged in. That is:
- Mushkin Stealth 991988S 4GB (U-DIMM DDR3-1333 LV)
- Hynix HMT112U7TFR8c0H9 1GB (ECC-DIMM DDR3-1333)
- Crucial CT51272BD1339 4GB (ECC-DIMM DDR3-1333 LV)
- Crucial CT102472BD160B 8GB (ECC-DIMM DDR3-1600 LV)
Note: the power consumption in UEFI setup is between 33 and 35 watts.

Power on

Firmware upgrade to P2.90

Now I decided to also upgrade the motherboard firmware to the latest downloadable P2.90. I downloaded the firmware onto a FAT32 formatted USB stick. Booted and pressed the [mono]F2[/mono] or [mono]Del[/mono] key to enter the firmware setup, choose 2nd menu "Advanced" and the lowest/last option "Instant Flash". The USB stick was recognised without any hassle, selected the firmware file and "Instant Flash V2009" started doing its job. That even includes a "Crashless feature". Firmware upgrades are painless.

Cold boot time

Boot time is dependant on the installed memory size. For cold boot time I do measure the time between "pressing the on" button and the time between "System Initializing 99" disappears and the bootloader (GRUB) menu appears. To exclude other factors all SATA drives are detached and System Setup is configured to not wait for the BMC. This measurement not an ice cold boot time measurement. The system/BMC is already powered, though not turned on.

Cold boot: memory size versus	boot time (in seconds)
1 × 1GB	19 s
1 × 8GB	22 s
4 × 8GB	33 s

Power consumption

Normal/idle

Ooh, I can't wait to find out what the real world energy consumption of this setup will be. Still with the Crucial CT51272BD1339 module, I pulled a few old and new SSD's out of the cabinet.

Normal/idle power consumption	P	Q	S	PF
Windows 8.1 enterprise, no HIDs	33.68W	47.77var	58.45VA	0.5762
Windows 8.1 enterprise, +blank screen	33.65W	47.51var	58.22VA	0.5780
Ubuntu 14.04.1, tty, OCZSSD2-1VTX60G	29,84W	44.00var	53.17VA	0.5613
Proxmox kernel 4.4.35, CT525MX300SSD1	31,70W	45.36var	55.33VA	0.5728
Windows 7 Pro, 850Evo Basic MZ-75E500	32,21W	46.38var	56.46VA	0.5704
CentOS 7, 3.10.0-327., CT525MX300SSD1	31,56W	45.30var	55.21VA	0.5717

Standby

Because the ASRock Rack C2X50D4I Series motherboard contains a BMC controller a.k.a. integrated lights out management a.k.a. Intelligent Platform Management Interface (IPMI) facility in the make and model of an ASPEED AST2300 IC, this machine will have relative heigh standby power consumption, even without any network cable attached.

Standby power consumption	P min.	P max.	S	PF
No ethernet cable attached	4.65W	4.77W	20.7VA	0.22c
Cable #1 = 7.5m AWG 26 Cat.5e (Inline 72575S)
1000 Mbps, Full Duplex, FC TX/RX	5.29W	5.36W	21.4VA	0.25c
EEE = 802.3az
1000 Mbps, Full Duplex, FC TX/RX, EEE	4.95W	5.03W	21.0VA	0.24c
100 Mbps, Full Duplex, FC TX/RX, EEE	4.76W	4.84W	20.7VA	0.23c
10 Mbps, Half Duplex	4.76W	4.83W	20.8VA	0.23c
10 Mbps, Full Duplex, FC TX/RX	4.69W	4.78W	20.6VA	0.23c
LAN1 add cable #2 = 5.0m AWG 26 Cat.5e (Inline 72505S)
2 x 10 Mbps, Full Duplex, FC TX/RX	4.83W	4.91W	20.7VA	0.23c

Power measurement setup

To find out how much power it consumes, I did already warm up the Zes LMG95 power analyser at least 30 minutes ago. This meter runs at 100 Khz, is roughly 0.025% accurate, and I did configure it to measure in 0.50 second intervals (that is 50,000 samples) and average 120 sampled intervals. This results in the average power consumption of a 60 second time window. Even with these averaging values, the measurements do drift in the 10's of milliwatts range over time. I could have increased the number of averaging measurement, but I did choose not to.
[img]afKnebk0[/img]

Power consumption tweaking

Aptio Setup Utility

Disabling Onbaord LAN 2 which I don't need in the near future. H/W Monitoring shows "Smart Fan Duty Control". Damn, the minimum duty that can be set is 30%, other options are 55, 60, 65, 70, 75, 80, 85, 90, 95 and 100%. Why am I not allowed to run at 0 % a.k.a. "off" when the temperature is below 45ºC. No bonus points here Asrock. Intel Speedstep, C Statte Support and Turbo Boost Technology are all enabled by default. Well done Asrock. Let's move on to another item of the "IntelRCSet" menu, the "Southbridge" configuration. Here the "SATA Link Power Managment" will be enabled for all ports. In "Server Mgmt" the "Wait For BMC" is now "Disabled" because I love fast booting. In the "Boot" menu I choose to not launch any optional ROM's (OpROM).

Normal/idle power consumption, tweak	P	Q	S	PF
CentOS 7, 3.10.0-327., CT525MX300SSD1	31.61W	45.80var	55.65VA	0.5680
+ PowerTOP Tunables toggled	30.48W	44.42var	53.87VA	0.5658
+ two SATA backplanes power unplugged	30.19W	44.09var	53.43VA	0.5649
+ tuned started	30.75W	44.65var	54.21VA	0.5673
+ tuned stopped	30.93W	44.85var	54.48VA	0.5677
+ tuned-adm profile powersave	30.29W	44.22var	53.60VA	0.5651
- 3 fans removed	27.29W	41.38var	49.57VA	0.5505
+ 11 Toshiba DT01ABA200 drives	69.56W	40.26var	80.37VA	0.8655
- 12 drives + Samsung MUF-32BB +Fedora25	27.70W	41.53var	49.92VA	0.5549
Fedora 25 + yum update	27.70W	41.51var	49.89VA	0.5550
+ yum install tuned; tune-adm powersave	26.82W	40.67var	48.72VA	0.5505
+ PowerTOP Tunables toggled	26.80W	40.63var	48.67VA	0.5507
- UEFI Marvell 9172+9230 disabled	27.36W	41.37var	49.59VA	0.5516
- UEFI Intel SATA 2 + SATA 3 co.disabled	27.18W	41.08var	49.25VA	0.5518
+ UEFI SATA Intel+Marvell enabled again	27.63W	41.28var	49.67VA	0.5562
+ 12 Toshiba DT01ABA200, 10 x standby	42.26W	54.52var	68.98VA	0.6126
+ 12 Toshiba DT01ABA200, 12 x standby	36.99W	49.90var	62.12VA	0.5955
+ 12 Toshiba DT01ABA200, 12 x sleep	36.92W	50.91var	62.31VA	0.5925
Crucial RAM 4GB->8GB, 3 x backplane unpl	27.11W	41.08var	49.22VA	0.5509
+ 8GB Crucial CT102472BD160B, tot:2xDIMM	27.40W	41.03var	49.34VA	0.5554
+ 8GB Crucial CT102472BD160B, tot:4xDIMM	28.01W	41.62var	50.17VA	0.5583
Enhance ENP-7025B PSU	28.56W	42.92var	51.55VA	0.5541
SuperMicro Platinum PWS-341P-1H	24.68W	42.09var	48.79VA	0.5058
Mean Well GST60A12-P1J + picoPSU-160-XT	21.28W	32.68var	39.03VA	0.5455
Enhance PSU + 3 x backplane powered, 12 x sleep	37.94W	51.20var	63.72VA	0.5954
+ 3 x backplane powered, 12 x sleep	37.94W	51.20var	63.72VA	0.5954
- 3 x CT102472BD160B, total: 1 x DIMM	36.89W	50.58var	62.60VA	0.5893

Hardware mods

Remove 3 fans

Nice thing about temperature sensors is that you actually can read the temperatures. With 45ºC as the lowest duty cycle and a CPU < 30ºC and a MB < 40ºC I started to wonder why do these fans even run? I can't turn them of in the BIOS, so how many fans this this system actually need? With a 250 watt power supply it can't produce more heat then for 250 watts that is obvious. How much will the system temperature then be above room temperature. Online I read that 18 degrees is a value that is used in thermal calculations. The PSU fan can move 8 CFM's though that is needed to move its locally generated 49 watts of heat (298W input minut 249W output). So I can't take that fan into account. Let's do the remaining math: 3.16 x 250W / 18º = 43 CFM. I did a quick test with 12 pcs of 5900 RPM SATA drives attached, then the whole system consumes roughly 80 watts / 100 VA at the wall socket. So only 83% of this will be the heat that needs to be adressed by the system fans. I dare to do this with only 1 fans, for some safety margin let's keep 2 fans. That means 3 buggers can be removed. That shaves of another 3 watts.

Move 1 fan

The "CPU Temperature" sensor reading is 25 º C and "MB Temperature" is 36 º C. What a high difference. Which sensor is "MB Temperature" anyhow? After installing the sensors package, it finds a Nuvoton (nct6776-isa-0290). Having a look at the motherboard/PCB shows a Nuvoton IC between DIMM and the PCI express slot. Let's proof it, move a fan exhaust air stream towards that IC and see whether the temperature drops. Damn, only 30 pictures are allowed in the Tweakers review format. Otherwise I would have shown my clunky setup that made the Nuvoton temperature drop by 3 ºC in 3 minutes. Why not permanently relocate the airflow there? Because the silver SATA backplane to motherboard cables are routed through that fan slot. An hour of hardware modding later the silver cables are routed through the fan slot that also carries the front panel cabling. Some carton cable protection has been added to the left and right side of the sheet metal edge.
[img]4gFDVJpV[/img]
An hour later and every cable is reconnected again.
[img]GW4SIsEs[/img]
And a few minutes later and the unused fanslots are filled with foam to optimize airflows.
[img]wRx1hQOT[/img]

Replace PSU

Despite that there are not many PSU's that will fit in 22 centimeters depth, I just wanted to know what it would do with power consumption. An 80plus Platinum Supermicro PWS-341P-1H was on hand. And with its cables routed through a chassis without cover, that Platinum PSU shaves of 4 watt in idle. However it increases standby consumption from below 5 to over 6 watts.
Despite its small dimensions, a pico PSU also will not fit within the chassis. Despite that, I was to curious what that PSU will do. So I ordered a "Level VI" efficient desktop 12V Mean Well GST60A12-P1J and attached that to picoPSU-160-XT. Idle power goes down from 28.5 watts to 21.3 watts. That GST60A12-P1J is underdimensioned for spinning up 12 drives, however Mean Well has got some bricks with more horsepower and even more efficiency in their product range: HEP-100-12 (92% typ.) and the RPS-200-12-C (93% typical).

Other issues

System lockup

During the last measurement I did remove the 3 Crucial DIMMs. After plugin them back in another time the system would no longer start. The boot screen was stuck at "System Initializing... 05". Let me be smart and use the BMC to "Power Off Server - Immediate", which was returned with a "Performing power action failed." dialog. After removing the power cord, removing all DIMM's, do a memory module hussle, and sticking them back in (probably in a different order) the system does boot again. Luckily hardware is often not exchange remotely ;-) so I won't make a fuss about this experiency.

Inrush current

While I measuring and having the 11 Toshiba HGST DT01ABA200 rotating drives and the Crucial SSD attached I was curious for the inrush current spike that is generated when turning on a machine with that many drives. For the number hungry, the Zes LMG95 measured Iinr at 3.0528 Amps.
[img]KdKqNIAl[/img]

Only 3 out of 4 USB ports are connected

During the installation of Fedora 25 Server, I noticed that I couldn't use all USB. I started reading the manual again. On page 19 there are the "USB Selection Jumper" and writes: Rear x 2, Front x 1 (Default). What the manual states as "default" is not the way the barebone came out of the box. After moving the 2 blue jumpers I finally had 2 USB 2.0 ports on the rear of the machine. Asrock improve your manual or building process. I did stick a piece of tape over the USB port that is not connected, to never attach a thing to the unconnected USB port. The picture shows the factory delivered USB blue jumper settings on position 2-3 (1 x rear USB 2.0 port support).
[img]6JmI1v46[/img]

Conclusion

To improve

Must have

- firmware that skips SATA disk detection (especiallly when not booting from SATA)
- a blank on the USB port not connected (match jumper setting)
- firmware setup that allows fan duty down to 0%

Should have

- an 86.0+ (Level VI) efficient PSU
- cable protectors in the metal openings where data cables are routed through
- blanks in the unused fan openings to improve air flow
- fewer fans or lower capacity fans

Could have

- engraved port labels on the rear side
- a lower capacity PSU
- a fan at a position that directs air towards the Nuvoton MB sensor
- undervolt capabilities for each PSU rail and several motherboard VRM's

Would be nice to have

- an ECO button that turns off all LED's, with a LED state readout and toggle in the BMC too
- 2 pcs dust protective blanks for the ethernet ports that are not in use
- a match between manual and factory default USB jumper setting
- a manual that tells to replace the lithium battery below 2.0 volts (in circuit)

Don't change

I love the full size (no SO-DIMM) memory slots. As well as the giant heatsink on the CPU that makes its operation fan-less (except for the chassis fans). I also like the Setup Utility option where the BIOS doesn't wait for the BMC to initialize. A nice touch are the tool-less screws for user changeable parts.

Alternatives

There are not many 12-bay 3.5" 1U barebones to choose from. Gigabyte has a 1900 euro (USD 1980) D120-C20 (6ND120C20MR-00) carrying a newer D-1541 processor and 4 bays extra. Supermicro has the SSG-5018A-AR12L which is not a barebone (it is a complete system assembled with memory, and 12 HDDs). That makes the Asrock cold storage barebone an easy choice.