general OD stuff

First of all, PLEASE use a tool like bender to do regular backups of your OS X Server services. It will backup your Open Directory configuration daily, and that can be a huge godsend if something goes wrong.

Open Directory is actually three different services wielded together. They are

1. OpenLDAP - for directory services, OD (Open Directory) actually uses a fairly standard OpenLDAP service. It has a few schema modifications, but other than that it is pretty vanilla!

2. Password Server - this is where your passwords and password policies live! Think of it like a black box. It gets backed up as well with bender, but there is not much you can try to do if it breaks (the mkpassdb command gives you a little bit of insight, but not much_.

3. Kerberos - for authentication/authorization/SSO type functionality. Again, this is actually a fairly standard install of kerberos. There may be a few modifications, but not many.

As an administrator, one of the biggest differences I noticed (having come from straight OpenLDAP on a linux server) was that OD does not have userPassword attributes for users. Instead, storage of passwords is farmed out to Password Server. The connection between ldap and the Password Server is negotiated with kerberos.
Long story short, this means that while you can think of OD like straight ldap, and in most cases you can use it that way, any integrations that want to scrape the userPassword attribute for authentication are going to fail - freeradius for instance (unless you are running it on the same server as your OD, in which case there is a module for that).

In my time managing Open Directory, I’ve had two major problems with it.

1. Kerberos issues - meaning I couldn’t authenticate with Password Server to make changes.
2. replsync errors. My replica wouldn’t replicate (which is a problem as I’m sure you can see).

My Kerberos issues went on for a long time - one day I went to change a user password and it just didn’t work anymore. Nothing had been changed on the server as far as I could tell - no new updates, no one had even logged in to the server since it had been working. I even had a ticket open with Apple for a while, but I never got any answers or resolution. Finally I restored from an archive (thanks bender).

The replsync issues were much more recent, and thankfully I actually managed to fix those. Basically I was seeing users, passwords, groupmembership, etc not syncing between my master and replica. The logs showed entries like these every minute or two (on both the master and replica):

Aug  4 09:53:13 ldap.example.com slapd[186]: conn=557025 op=3: attribute "entryCSN" index add failure
Aug  4 09:53:13 ldap.example.com slapd[186]: do_syncrep1[done]: si_ld(0x0) Can't contact LDAP server (-1)
Aug  4 09:53:13 ldap.example.com slapd[186]: do_syncrepl: rid=003 rc -1 retrying

So what was going on exactly? I tried destroying the replica and setting it up from scratch. That sort of worked - I got an up to date replica, but the errors in the logs didn’t stop, and any changes I made on the master still would not replicate.

The servers could obviously see each other, or else the replica couldn’t be created in the first place.

The next tip I followed led me here. I did have a _ldap_replicator user, but maybe the password had gotten out of sync somehow? I made use of the mkpassdb command to try and generate a new password for that user, but that didn’t seem to help either.

So I was forced to look elsewhere. I ended up figuring out the issue was in the cn-authdata records.

Here is the process that fixed it:
On master and replica:
slapconfig -stopldapserver

On master only:
db_recover -v -h /var/db/openldap/openldap-data/

On master and replica:
slapconfig -startldapserver