This episode gives credit and relies heavily/completely on the presentations, notes and speeches done and copyrighted fully of Dr Marshall Kirk Mckusick who is a renowned computer scientist, known for his extensive work on BSD UNIX, from the 1980s to FreeBSD in the present day.
Much of the information in this shownotes can be found in the chapter on Berkeley UNIX (pages 31-46) in the book: Open Sources: Voices from the Open Source Revolution
If You are interested further as this episodes and shownotes are a very stripped down version of the work of Dr Marshall Kirk Mckusick´s notes and presentations Please consider purchasing His DVD of the History Of Berkeley Software Distributions from which this episode excerpts its facts and figures and historical recounting.
It is a near 4 hours long 2 parts presentation which recounts the History of BSD from the early days and another lecture about the Modern FreeBSD.
It is great fun to watch and own and definitely a better recount of the events with many fun facts and personality as I could ever do myself.
Parts which I left out from the History can be found on Kirk McKusick‘s History of BSD DVD release.
History of BSD
The Atlas Supervisor
One of the early time sharing and modern operating system of its time from the 1960s from Manchester University’s Machester Project.
It ran on the Atlas and Atlas-2 Titan computers
It set up many of the principles of what time sharing would actually mean
After World War II. many of the scientific minds came over to the United States as there were far more research opportunities even into the 1960s
A Big Project gets started in the US – Multics
A joint project between MIT, General Electric and Bell Labs
- GE was going to provide the special hardware *at the time it was thought special hardware was needed to do time sharing properly* included rings of protection which evolved into todays supervisor mode and user mode
- MIT provided academic input
- Bell Labs provided the industry support whereas Kirk McKusick described this as he believes they provided much of the funding of the project
The Project was called Multics
It ran on the GE 635 * it was considered to be a mainframe at its time, GE is no longer in the computer industry*
There were of course issues around Multics. It was driven by several groups *MIT, GE, Bell Labs*
Many of the participants had good ideas as to what time sharing should be they d come up with an idea prototype it and when they were convinced it was workable …. they moved onto some other idea … Resulting in the fact that Multics never really became a finished working product or an operational system in the sense of the word.
This resulted of Bell Labs eventually abandoning the project as they wanted a working time-sharing system they just got tired waiting for it to be finished
Bell Labs people who returned from the Multics project they were stuck to use GECOS * General Electric Comprehensive Operating Supervisor* originally a batch processing OS for mainframes. It was a letdown after the interactive time sharing… as this one required punch-card inputs. 🙁
Two of these guys Dennise Ritchie and Ken Thompson were not about to work on the GECOS system. They have already saw what an OS like Multics could be *time sharing OS*
They started to work on an abandoned PDP-7 ( late 1969) a minicomputer from Digital Equipment Corporation.
They got to the point where they developed what we would call today as an embedded system for the PDP-7 so they could run games on it. They were determined that they needed a more powerful computer to do what they wanted
Computer Science Department of Bell Labs had nowhere the budget and structure what it has today therefore they approached the Legal Department which had a bigger budget (1971)
They struck a deal with them about the following:
- Legal Department would purchase a PDP-11/20 machine
- The Computer Science Department writes a program to help the Legal Department with text processing.
This led to the development of roff which was the UNIX version of Multics‘s runoff program .. later became rewritten as nroff
UNIX and C Programming language
They started to work on what eventually became UNIX * Ken Thompson started working on UNIX while Dennis Ritchie was working on the C Language*
Originally written in Assembly language but later rewritten in C language
They presented UNIX at their first talk about the operating system at the ACM (Association for Computer Machinery) Symposium on Operating System-Principles in 1973.
They were discussing UNIX Version 4 at the conference ( presumably the C language version but there is no accurate confirmation of this fact – i have reached out to Ken Thompson to get a confirmation on the same and I will update this article if it ever happens )
UNIX struck a nerve with academics it looked cool it gave them the light in the end of the tunnel to break free from Mainframes ran by central authorities of the universities and its servants of dead bodies aka terminal computers governed by the computer administrators.
Buying a PDP-11/20 for an academic department seemed like a realistic goal or obstacle to overcome even with a slight budget stretch VS ever dreaming of owning a Mainframe for their department they can govern above.
On this first conference Bob Fabry was present a now retired professor of UC Berkeley who wanted UNIX and all those utilities and lucky for him Ken Thompson was an old alumni of UC Berkley so it gave him the inside track. It resulted the Computer Science Department of UC Berkeley outside of Bell Labs to get UNIX Version 4 which shortly after was upgraded to Version 5.
The Berkeley Computer Science department was not happy running on PDP-11/20. They wanted to run on a newly available PDP-11/45.
They did not have the money to buy one on their own so they made a deal with Math and Stats Departments around 1975
But there came a compromise. The Math and Stats department wanted nothing to do with this new experimental UNIX System. They wanted to use a real operating system called RSTS which was DEC´s operating system for the PDP-11.
This resulted in rotating timeframes ( 3x 8 Hour periods):
- noon to 8:00pm
- 8:00pm to 4:00am
- 4:00am to noon
UNIX was run in rotating timeframes:
- noon to 8:00pm one day
- 8:00pm to 4:00am the next day
- 4:00am to noon the day after that
RSTS ran for the remaining 16 hours of each day
UNIX users were chasing the UNIX time slot around the clock until exhaustion then they would get caught up on their sleep and start chasing the UNIX time slot again.
Many people wanted to run UNIX so around 1974 the Computer Science Department of Berkeley bought a PDP-11/40 which was running the latest Version 5 of UNIX 24 hours a day and it worked well minus one hour a day for panics crashes and reboots…
Professors wanted to teach classes on the UNIX machine. It was better than having the graduate students punching cards but the PDP-11/40 does not have the power to support such use case but it opened up the possibility to use ,,instructional use” funds from the state of California to be used to purchase a PDP-11/70 the latest machine from DEC at the time with a one MIPS processor (in 1975)
They purchased the machine with two disk drives, it had only a single disk controller
The fact that UNIX in theory was able to issue two seek commands one each on the two disk drives the seeks could overlap in time and on whichever disk the seek completed first UNIX could start the transfer. A transfer for the other drive would be started at a later time. Only one transfer could occur at a time.
Previously at Bell Labs they did not stress tested parallel seeks. This computer at Berkeley was bigger with a load average of 40 which stress tested this issue for them plenty.
As a result the system experienced strange hangs and panics.
Ken Thompson had written the relevant disk software therefore Berkeley called Ken for help.
He dialed in from New Yersey and Berkeley connected him to the machine via a 300 baud acoustic coupler modem. (Probably the first instance of remote debugging in UNIX)
Ken Thompson figured out the issue quick and doing so peaked his interest and as he was due a sabbatical leave he decided to spend the year at Berkeley over the 1975-1976 school year.
Bill Joy, Ken Thompson and the “50 Changes” Tape
At the same time Bill Joy matriculated at Berkeley (Fall, 1975)
Bill Joy was interested in programming language research and Pascal had just arrived on the scene and was about to take over the world.
Previously Ken Thompson put together/hacked together a Pascal thing a byte-code interpreter ( a compiler for a subset of Pascal which compiled down to byte-codes)
Bill Joy worked on this and turned it into a real Pascal Compiler (not a full compiler tough but it compiled Pascal down to byte codes which were then interpreted) He added better error handling and recovery which allowed students to address a number of errors at once
This allowed Bill to learn UNIX and it also allowed him to work with Ken Thompson, Ken was administrating the system and also actively developing the kernel.
Bill also learned other important things f.e its a good idea to do dumps , use dump (8) command, used to make a backup of a file system, disks do fail occasionally and people like to be able to recover their files.
Bill stepped into the role of system administrator after Ken left from his sabbatical
After Ken Thompson returned to Bell Labs from his sabbatical He put together the “50 changes” tape which were 50 patches to apply to Your UNIX kernel to make it work better.
Bill Joy obtained the tape and applied the patches
He then rebuilt the kernel (this is how he learned to build kernels, etc.)
There were no autoconfiguration or autorestart or things like that back in that time.
The kernel had to be compiled exactly for the hardware it was to run on.
You had to keep two kernels around:
- one was compiled for one less disk than You had and You would boot this if you lost a disk
Bill did a lot of editing while doing this work. He found that the ed editor had a lot of deficiencies , He found em editor written by George Coulouris of Queen Mary`s Colleague London.
Bill Joy had a rule (his first rule): never spend a lot of time coming up with a good idea when You can steal a better one.
End of Part I. – Episode 30.
Start of Part II. – Episode 31.
Bill Joy then hacked on the em editor and came up with the ex line oriented editor.
Berkeley at that time did not have any screen oriented terminals * They had ADM-3 line-oriented dumb terminals*
He was very good at deciding what the problem was and finding the shortest path to solve it.
Bill now had two interesting software systems He had put together: the Pascal interpreter system and the ex editor.
He combined these into the Berkeley Software Distribution shortened for BSD later still called 1BSD. He distributed about 30 copies of this on tape starting February 1977.
He was very good in multitasking: being a grad student, doing distributions, sending out mails, answering the phone, hacking on code ….
Berkeley after a while did obtain some ADM-3A screen addressable terminals: it meant that You could move the cursor anywhere You wanted on the screen. Bill took his ex line oriented editor and converted it to a screen editor. And that is how the Vi editor was born.
However He did not want to create 3 different versions of his Vi editor to handle ADM-3As , ADM-3s and DEC Printing terminals. so He came up with “termcap” (terminal capabilities)
Isolated some of these characteristics into strings , dynamically configured the editor based on the terminal You were using.On the downside this allowed terminal manufacturers to create any strange screen addressing they wanted knowing they can write a termcap entry to make it work.
It resulted the termcap file to grew into a monster in size.
Bill finished working on Vi however it was never actually finished it was just good enough for what he and the people around him had to do with the editor.
He did not like the bourne shell ( written by Stephen R. Bourne while at Bell Labs) and the way it worked He wanted something more C like so He started working on the C Shell. Bill was a graduate student in programming languages.
Meanwhile He did an update to the original Berkeley Software Distribution. He called it 2BSD.
He shipped about 75 copies starting in June 1978
2BSD was the last distribution done by Bill Joy for the PDP-11
There was a continued interest in PDP-11s, so other people started backporting the work Bill was doing on the VAX to PDP-11s which led to the releases such as 2.1BSD, 2.2BSD, etc. eventually leading to the 2.11BSD release which still ran on PDP-11s well into the 1990s.
- By 1978 the Berkeley Computer Science Department has begun to make a name for itself which as per Kirk McKusick was thank to Bill Joy and his work. As a result they had enough grant money to buy the latest machine from DEC a VAX 11/780 delivered in 1978 was one of the first ones off the assembly line.
They wanted to run UNIX on this machine. They obtained a copy of UNIX/32V a fast and dirty port of UNIX Version 7 for the PDP-11 ( no virtual memory , no paging) which limited them to 128K in size just like on the PDP-11 , limited but it got the VAX up on UNIX.
- The PDP-11 was limited to 128K text + data
- The VAX was limited to the amount of physical memory on the machine, minus the memory dedicated to the kernel
- the department VAX 11/780 was fully loaded with 2Mbytes of memory
so, the user process was theoretically limited to about 1.5Mbytes
The people who had spent a lot of money on the VAX did so because they wanted to run vaxima a differential equation solver written in LISP.
in those days:
- when you typed “lisp”, as soon as you got a prompt,
you were using a megabyte
- as soon as you typed anything, you were up to two
- as soon as you started doing anything, memory usage
went up from there
Unfortunately, this machine had only 2 megabytes of memory, users could not do anything useful in LISP. They needed virtual memory. VAX/VMS operating system had it
Luckily for them Bill Joy comes to save the day and tells them He could get Virtual Memory working in UNIX on the VAX. And He said he could do it over the four week Christmas break.
He was given until the first day of classes in January else DEC‘s VAX/VMS would be used
Remember the first rule of Bill Joy?
never spend a lot of time coming up with a good idea when You can steal a better one.
Ozalp Babaoglu had hacked together a Virtual Memory system for his PhD thesis. Bill took this and hacked it into UNIX. He was not concerned by the fact that Ozalp‘s Virtual Memory system was not finished.
UNIX/32V and Virtual VAX/UNIX or VMUNIX (Virtual Memory UNIX) were up and down alternatively over the course of the break period. VMUNIX was up more and more as the weeks went forward and was up by January 18th.
Was a bit rocky for the next couple of weeks. LISP now worked but slowly however it was not the fault of UNIX. Users were trying to solve differential equations using 10MB of virtual memory on 2MB of real/physical memory.
With this newfound success came the necessity to port all of the utilities from 2BSD to the VAX
By this time Kirk McKusick shares an office with Bill Joy and He is drawn more and more into the works of BSD.
Once all the utilities were ported Bill Joy and his office peers (Peter Kessler, Robert Henry, Kirk McKusick and a few others around the department) had their first complete system:
a kernel, the utilities, everything you needed to load onto a bare metal hardware and just run it.
They released it as 3BSD
As a reminder from 1984-1996 Bell Laboratories is called AT&T Bell Laboratories.
Bill Joy is still a graduate student still hacking code and doing distribution but now a new problem is rising on the horizon. the distribution 3BSD includes also 32V code (UNIX/32V from AT&T Bell Labs)
Previously the distribution just included code that had been written at Berkeley now that it included 32V code licensing was in need.
Bill Joy took all the calls which came into the shared office.
License verification was just a verbal confirmation.
About 100 copies went out under these terms of 3BSD starting in December 1979
This 100 copies meant about a 100 different organizations resulted of each copy deployed on multiple systems. Major universities , a few companies and research groups.
As this system 3BSD is more widely distributed a lot more people begin to get involved in it.
Initialy a 32V license cost 99$ which rose to around 10 000$ later.
( I included two links from original documents scanned from 1983 – 1984 showing UNIX System V license prices it is interesting to see the astronomical rise those UNIX license fees went through )
This is how the 3BSD and Licensing worked in this scenario:
– The organization would buy a VAX
– purchase a 32V license for it
– and then run 3BSD on their system
As a result of the wide distribution, more people started hacking on the code resulting in more contribution to come in.
Auto reboot also known as auto restart became available around this time. When a machine crashed it could automatically reboot itself.
Previously when the machine halted someone had to restart it from the console. It was in part made possible because of the first versions of fsck which meant that after a crash there were no need anymore to a computer guru to manually run filesystem check and cleanup commands on the file system to clean it up. It finally became to be possible to operate 24/7 which meant the systems could go into heavy production.
One of the early LISP systems was coming out around this time.
Also thanks to a program called delivermail the predecessor of sendmail both created by Eric Allman it became possible for the first time to have email travel between two different machines.
When Eric Allman rewrites delivermail into sendmail he decided that You shouldn’t have to recompile your whole email delivery agent every time You wanted to change something. Delivermail had everything compiled into it.
This resulted in the sendmail configuration file so you could reconfigure it on the fly without recompiling th ewhole thing over and over again each time. He also made it work with the SMTP Internet Protocol.
All these changes mentioned above plus improvements to the File System and some other bits and pieces were gathered all together and released as 4BSD starting in October 1980
About 150 copies of 4BSD were shipped however not all had to be handled by Bill Joy.
The University noticed that something big was happening out of this graduate student’s office.
This resulted in the University lawyers to start to sniff around and they decided that license verification was not rigorous enough. 🙂 who d have thought 🙂
An administrative assistance was hired to do license verification.
Many of these above activities became possible because of a new source of funding became available actually driven by VAX VMS the funding was coming from DARPA (Defence Advanced Research Agency)
DARPA as an arm of the Department of Defense was coordinating the research needs of the various arms of the military (Army,Air Force, Navy, Marines, etc.) was funded by these various branches and it was supposed to determine which projects were worthy of funding and which branch of the military should fund which project.
Projects were supposed to have some bearing on military readiness.
The situation with the military was:
– they had many projects going on
– they were running on different hardware
– they were using different operating systems
– they were using different programming languages including assembly language
– they were all using legacy, propietary systems
This resulted in that nobody could share results or programs with one another
DARPA was going to decide on:
- a certain computer
- a certain operating system
- they would get everyone they were funding to use this system
- so when they developed something they could share it with the other groups/branches
The machine with the best price/performance for the size of grants they were dealing with was the VAX.
The only remaining question was: Which operating system to use? VMS or UNIX?
VMS was the initial favorite. It was written by the vendor, it was supported by the vendor and clearly You could get support if You had issues. However the researchers wanted UNIX.
David Kashtan decided to resolve the VMS Vs UNIX matter
He was then at Stanford Research Institute , He was also a DARPA grant recipient interested in helping to select the hardware and OS to be the basis for the standard.
He wrote the famous VMS paper where he described the results of benchmarks he ran on both VMS and Berkeley UNIX systems.
These benchmarks showed severe performance problems with the UNIX system.
He developed micro-benchmarks:
- how fast can you do getpid()?
- how fast can you write a byte back and forth over a pipe between two processes?
to measures context switch times
He ran these micro benchmarks on VMS and on UNIX.
Performance was better on VMS in all cases
So because of performance measurement and vendor support He concluded they should standardize on VMS.
Bill Joy received a copy of David Kashtan‘s paper.
He decided to speed up the benchmark cases in UNIX
He did a lot of tuning:
Created specialized code, optimized the assembly language, etc.
Was able to make all the benchmarks run faster on UNIX than VMS except for the context switching.
Through surreptitious means Bill and his associates were able to determine how VMS did context switching which resulted UNIX doing the context switching benchmark the same speed as VMS did.
Bill Joy then wrote a rebuttal paper. He trashed the validity of the benchmarks but he then showed that UNIX was as fast or faster in all the benchmark cases.
This resulted in DARPA funding work at Berkeley began about June, 1980.
Bill were commissioned to ship the work they had done to make UNIX so fast (the performance tuning and code optimisations). As these were made as proof of concept some time was needed to make it production ready.
Autoconfiguration had to be added so that BSD recipients did not have to know how to build a custom kernel.
Bill Joy added additional things including autoconfiguration written by Robert Elz at the University of Melbourne, Australia. The system could look around at the hardware present and configure it therefore specific configuration did not have to be compiled in * to the kernel*
The performance tuning and autoconfiguration came out as the next Berkeley release 4.1BSD
People at AT&T were concerned that a name like 5BSD could be confused with System V, hence the switch to using the 4.X as the name of the release.
Around 400 copies of 4.1BSD were shipped starting in June, 1981
At the time AT&T was shipping the following:
- UNIX Version 7
- Programmers Workbench (PWB)
- System III Created in early 1981 which was a combination of Version 7, 32V, PWB, USG (UNIX Support Group) Tools
None of these offerings ran particularly well on the VAX, none had virtual memory
Therefore AT&T would sell a System III license which they got very good money for and the buyer then get a tape from Berkeley with 4.1BSD to run on the VAX
DARPA was pleased that Berkeley shipper their release (4.1BSD) in only a year however it was promised in four months. It was still very fast compared to other DARPA contractors.
As a result they decided to award another contract to Berkeley, This time to get networking into UNIX. UNIX already had networking of a sort with UUCP (Unix-to-Unix Copy) but DARPA had bigger plans.
DARPA did not trust a university / academics to do something as critical as implementing a protocol stack for example something as complex as TCP/IP.
DARPA decided Berkeley to design:
- an API for accessing the networking
- create device drivers for Ethernet controllers
To implement TCP/IP protocol stack DARPA contracted BBN (Bolt, Beranek and Newman)
DARPA set up a steering committee to oversee the development.
In response to the DARPA contract Bill Joy wrote the 4.2BSD architecture manual.
It took him a couple of weeks and it described everything that would be in 4.2BSD (including things like mmap which was not implemented for another eight years)
Everything was to be implemented in one year.
The manual included the networking interface, included function prototypes of the socket interface, accept(), connect() , etc. at the level of man pages. It gave descriptions of how the system calls would be used.
Bill then asked BBN for a copy of their code so he could test his interface.
Rob Gurwitz from BBN provided a pre-release version of their networking code to Bill.
The environment where Rob Gurwitz made BBNs networkign code consisted of mainly VAX 11/750s and 56Kbit lines. His network code saturated a 56K line with 100% of CPU use on a VAX 11/750.
At Berkeley however they had 3 Mbit ethernet the latest and greatest from Xerox PARC
Doing bulk data transfer between two machines connected with this 3 Mbit line using BBNs code the throughput was 56K.
This resulted in Bill Joy hacking up the BBN code completely. As a result he was able to saturate the 3 Mbit ethernet using only 90% of the CPU
Many people wanted to start testing the interface so this system was informally distributed as 4.1a BSD starting at April, 1982. It was basically 4.1BSD plus the networking code, actually an alpha release but it proliferated widely.
At this point Sam Leffler joined the group. He came from a company that had a networking product. He had an advanced degree in computer science.
He saw deficiencies in the networking interface which he corrected.
Summer was approaching and Kirk McKusick asked Bill if He could work on some project during the summer while He also worked on his thesis.
Without getting into details ( for the little nuances and anekdotes on how the FFS filesystem came alive You have to watch Kirk‘s DVD of History of BSD) nevertheless this little side project of Kirk‘s resulted in 18 months of the FFS (Fast File System) which took Bill‘s original prototype filesystem into a release-ready filesystem which went into the 4.1b BSD Release in a form of test distributions in June , 1982
As a result of the new Filesystem (FFS) and the 4.1b release , Bill Joy funds Kirk‘s trip to USENIX conference in Boston with the help of the DARPA funds where He went on stage one of the first times , talked about the Filesystem and as You can see thanks to his lectures and presentations He never really gave up ever since.
The team at Berkeley had put together their UNIX system.
Bill Joy and Sam Leffler had revised the networking and the IPC Code (inter-process communication)
Bill suddenly became interested in separating the code into machine-dependent and machine independent parts.
Later on He shared it with Kirk that He was thinking of going to a startup called Sun Microsystems
The idea was the following:
They were going to take commodity Motorola 68000
microprocessors, and run BSD UNIX on them
as a side note:
a number of small companies were selling 68000-based
boxes already, but running a variant of UNIX from
Santa Cruz Operation (SCO)
After Bill arrived, Sun was were able to
ship BSD and as Per Him BSD UNIX “will be so much better”
The marketing pitch was:
- Open systems
- Commodity hardware,
- Commodity software,
- If your vendor is not satisfactory, you can go to whomever is cheaper or better
- People will buy into this,
- Sun will sell a lot of systems
Bill tried to convince Kirk to join as well.He could get a single-digit employee number and great stock options at Sun.
Eventually Kirk McKusick did not join to Sun He had his own reasoning behind it which He explains in detail in his presentation on History of BSD
After 18 months Kirk finished his PhD, by then Sun was a big company.
He did however was the first hired consultant at Sun by Bill to port the Pascal compiler which was written in assembly language. Kirk completed the work together with Peter Kessler.
Personally I would have loved to see Kirk McKusick joining to Sun.
Perhaps there is an alternate Universe out there where It has happened where Sun thrived through the 2000s An alternate Universe where Oracle does not exist and where Sun , DEC, Compaq still the Big Names in IT.
End of Part II. – Episode 31.
Start of Part III. – Episode 32.
Bill Joy decided to do a test release called 4.1c, just as He was leaving Berkeley to go to Sun.
It contained the FFS( Fast File System), the networking code, the new signal work that had been done written by Sam Leffler. The release came out in April, 1983.
The initial copy went to Sun Microsystems, later some copies went to other users.
There was talk of release 4.1d but eventually it never happened. It supposed to be the VM (Virtual Memory) release with mmap. Bill Joy was committed doing this release before he fully transitioned to Sun. But He was gone before it could happen and also Bob Fabry was gone the professor who was running the project.
This is the end of the Bill Joy era of Berkeley Software Distribution
There was significant pressure to release a system, it had already been a couple of years since 4.1BSD had come out.
Mike Karels joined the project around June,1983 from Molecular Biology.
Sam Leffler and Mike Karels with some help from the rest of the team put together the 4.2BSD release released in August, 1983
(AT&T System V was coming out in about the same time frame)
About 1 000 copies of 4.2BSD release were distributed, one copy per site , a site represented all the machines at a major university or all the machines in a large corporations in another words it represented a huge number of machines.
4.2BSD represented a very important release a high point of Berkeley distributions in terms of the commercial world.
At that moment You could have:
- System V with UUCP
- OR You could get 4.2BSD with TCP/IP
- System V with around 30 KBytes.second filesystem bandwith
- OR 4.2BSD with 400KBytes.second of filesystem bandwith
There was such a high demand for 4.2BSD the Berkeley team could not make release tapes fast enough to meet demand. Also AT&T was not in a hurry to verify licenses which slowed the whole process down as well.
The first seven 4.2BSD tapes had to go to seven designated DARPA recipients.
One of them was BBN ( Bolt Beranek & Newman)
—– TCPIPWars ——
The continuation of the story would follow with the TCPIPWars which I will leave out and continue from after that happened.
—– TCPIPWars ——
Berkeley issued the 4.3BSD release in June, 1986.
Bill Joy had left by then fully.
The Team at Berkeley started to restaff the project.
Kirk McKusick joined the project in January, 1985
Keith Bostic joined October, 1986.
Keith had a requirement for taking the job:
He needed to be allowed to finish his port of release 4.2BSD to the PDP-11 which would become 2.11 distribution
Kirk and Mike agreed to this in his spare time.
Keith Bostic’s job was divided into three parts:
- 1/3 of the time He would answer the phone
and provide technical support their users
if it was something interesting he would work with them
particularly if they had a bug fix in hand
- 1/3 of his time he dealt with bug report emails
- 1/3 of his time he was to do software development
- And he worked on the 2.11 project as well for the PDP-11 in his spare time as said.
He finished the 2.11 release in late 1988 ( about 2 years after He joined the Team)
The next release after 4.3BSD was called 4.3-Tahoe ( there was a debate at that moment if that release should have been called 4.4BSD) released in September, 1988
Mike Karels was of the opinion that 4.4BSD should have a certain set of features which this release did not have to be called 4.4BSD
The primary purpose of this release was to support a second architecture
** some partitioning of the code was done at the time Bill Joy was leaving mentioned previously **
This was the first full fledged port to a new architecture, ported to a computer called Computer Consoles Inc (CCI) 6/32 machine which had a similar architecture to a VAX but 5x faster with the size of a large deskside machine
Meanwhile System V (AT&T) and commercial OS vendors continued in putting out ever more complex systems including networking , virtual memory, etc. but the effect of this was that the price of these operating systems and the systems shipped with these operating systems kept increasing.
Kirk McKusick notes that the only real commercial UNIX was still System V at that time
Many users wanted to get the BSD code from Berkeley but they could not afford the $250 000 source license from AT&T so they could pull out / would be allowed to pull out the TCP/IP stack and use it in some embedded application.
This raised the question:
Since the TCP/IP stack code was developed at Berkeley, could not it be released separately?
[DARPA placed no restriction on code releases, other than requiring that the code be distributed to other DARPA contractors]
All DARPA contractors had UNIX licenses so it was not an issue.
The socket interface and the TCP/IP code was done entirely outside of AT&T
Therefore it contained NO AT&T proprietary code so people who wished to incorporate it into products should not have to buy an AT&T license to use it.
The above led to the release of “Networking Release 1” “Net1” or “Net/1” released in June, 1989 which included the following code extracted from BSD release:
- network device drivers
- TCP/IP stack
- the “R” commands: rcp, rlogin, rsh, etc. which were quick hacks created by Bill Joy to test the networking code and originally meant to be temporary until the real commands (ftp, telnet, ssh, etc) could be written –> the R commands persisted for two decades
They used trusted ports only <= 1000 that could only be opened by processes running root
A 9 track tape of this code cost $1000 however it was available over anonymus UUCP for free as well many people bought the tape , they wanted the piece of paper saying this was freely redistributable
The Berkeley team got back on their main project , continuing to put in all the features that were supposed to be in the final 4.4BSD release
They decided to do another interim release, a fair amount of time has passed since the last release.
4.3-Reno was the name planned for the next release partly named because Reno was a gambling capital and it was sort of a gamble to run this distribution as not much release engineering had been done.
It contained a number of major new systems:
a new virtual memory system (previous one has been developed by Bill Joy and Ozalp Babaoglu back in 1979)
Kirk McKusick was put in charge of the new VM system work started in 1988 and first became operational in late 1990
He had two candidates for the new VM system that looked very good
Sun had done a VM system that was written straight to the architecture manual interface
* the 4.2BSD architecture manual , written by Bill Joy in response to the DARPA contract previously)
CMU had done the Mach Project (Carnegie Mellon University) part of a microkernel project , could be used without taking the microkernel part. DARPA favoured this option.(the MACH)
For the Berkeley team to use the Sun VM system , Sun had to release it. They had a discussion about this around 1988 where even tough every major player of Sun including Bill Joy and Scott McNealy *co-founder and CEO of Sun Microsystems) including technical people and their managers agreed it was the good idea… the board of directors and lawyers advised against it * could be taken legal action against Sun by its stockholders giving away company property for free*
As a result of the Sun decision the Berkeley team went with the Mach VM system.
Mike Hibler at the University of Utah did the work of integrating the Mach VM system into BSD then Kirk McKusick made minor changes.
At the same time the Berkeley team wanted to put NFS into BSD * Network File System originally developed by Sun Microsystems)
NFS was taking over due to Sun‘s skillful marketing
Sun had placed the specification into the public domain.
Rick Macklem at the University of Guelpf did the work in Ontario, Canada during the long winter Rick had written NFS.
The 4.3-Reno release started shipping in November, 1990 and it included the following:
- The New VM system ( Kirk McKusick says this came not in the 4.3 release intermediately but very soon after it came in an updated 4.3-Reno release)
- the new vnode stuff which were updates to the vnode interface to support the addition of mmap. This was written by Kirk McKusick
Meanwhile the release of Net/1 led to a desire to release more of the BSD software freely.
Calls were coming in to Keith Bostic requesting this who brought it up every week in the Berkeley team’s weekly meeting
Kirk and Mike pointed out impediments to the release:
- not just a kernel issue
- there are many utilities
- there is the C library
- all of which are riddled with 32V code
it would be a mammoth undertaking to sort all this out
Kirk and Mike told Keith that he could work on this in his spare time if he wanted to
They figured this would be the last they would ever hear of this issue
At the next USENIX conference, Keith gives a presentation:
He puts up a list of utilities “I need people to write these utilities”contribute them to Berkeley, get recognized for their accomplishments.
People started to rewrite certain simple to relatively complex utilities ( cat, od, head, tail)then someone rewrote troff provided evidence that BSD contributors were serious
Keith was busy integrating the contributed utilities, he was also busy rewriting parts of the C library , one day, Keith ambled into one of the Berkeley team meetings he had the C library rewrite mostly done, he had about half of the utilities done.
He asked Kirk and Mike how the kernel was coming
Kirk and Mike realized they couldn’t get the user community to rewrite the kernel
Kirk, Mike, and Keith built an inverted database from the
32V source code they then went through the BSD source code, line-by-line,
looking it up in the inverted database to see what matched
When the code inspection effort was over, they found that
only about six files were contaminated
(note that trivial similarities had already been rewritten as a part of this effort)
They thought that they could rewrite the six files
But then they thought it would be better to release the software without the six files
Without these 6 files The software would be broken, it wouldn’t even compile, so, hopefully, AT&T wouldn’t even notice
The Berkeley team felt the need to talk to the higher-ups at the University they felt they shouldn’t do this release on their own they talked to the University lawyers about the license to streamline the process, they reused the Net/1 license
with a name change to Net/2
The Berkeley team talked to the head of the Computer
Science department they escalated it up to higher levels within the
University to the Office of the President of the University of
(the Office of the President oversees all the UC campuses)
Auditors were brought in,the Berkeley team spent three days being audited
Permission was finally granted from a very high level of the University
The Net/2 software was finally release in July, 1991
The software was well-received many people bought it, as before even though it could be downloaded for free via ftp.
By early 1992, several groups had figured out how to
rewrite the six missing files
The Berkeley Software Design Inc. (BSDI) company was shipping an alpha version of their product by January, 1992
Bill Jolitz had rewritten the files, he had released a system called 386/BSD
About this time, Mike Karels decided to work at BSDI (Berkeley Software Design Inc.)(BSDI, later BSDi)
Note: Kirk McKusick was an agel investor in BSDi
BSDI had been shipping a product, They ran ads with the phone number: 1-800-ITS-Unix
AT&T sent a cease and desist letter to BSDI
Stop shipping the product, or get a license from USL
(UNIX System Laboratories a mostly-owned (majority-controlled) subsidiary of AT&T
(80% owned by AT&T)
BSDI was a four person startup
USL filed a lawsuit against BSDI which led to a Lawsuit which I will leave out
—– Lawsuit ——
The continuation of the story would follow with the Lawsut which I will leave out and continue from after that happened.
—– Lawsuit ——
The original intent was to release two versions of 4.4BSD.
4.4BSD also referred to as 4.4BSD-Encumbered which had everything and required an AT&T license and 4.4BSD-Lite that had only freely redistributable code.
Eventually 4.4BSD-Encumbered (4.4BSD) got released in June, 1993 being tired of waiting for resolution of lawsuit.
4.4BSD-Lite was released in June, 1994
Keith Bostic and Kirk McKusick continued working at the University putting in bug fixes and enhancements as they were received for 4.4BSD and 4.4BSD-Lite
As a result They released 4.4BSD-Lite2 in June, 1995
It was the last BSD distribution that came out of Berkeley
From the article of Charles Babcock from Information Week published at the 8th of November, 2006
The single Greatest Piece of Software Ever, with the broadest impact on the world, was BSD 4.3. Other Unixes were bigger commercial successes. But as the cumulative accomplishment of the BSD systems, 4.3 represented an unmatched peak of innovation. BSD 4.3 represents the single biggest theoretical undergirder of the Internet. Moreover, the passion that surrounds Linux and open source code is a direct offshoot of the ideas that created BSD: a love for the power of computing and a belief that it should be a freely available extension of man’s intellectual powers–a force that changes his place in the universe.
DVD order of Dr Marshall Kirk McKusick
Order RunBSD Stickers
Kirk McKusick Homepage:
BSDTalk Interview with Kirk McKusick
Wikipedia Entry on the History of Berkeley Software Distribution: https://en.wikipedia.org/wiki/History_of_the_Berkeley_Software_Distribution
Bell labs on wikipedia:
UNIX License Fees (1983-1984)
UUCP – Unix-to-Unix Copy
4,2BSD Networking Implementation Notes:
4.3BSD Virtual Memory Management (with some comments on 3BSD)
Harris / Tahoe Platform:
What’s The Greatest Software Ever Written?
4.4BSD Lite Release 2: last Unix operating system from Berkeley
(tons of documentation links on the Github page)
SIMH Compatible Tapes of BSD Releases:
The UNIX Heritage Society:
Run 32V 3BSD and 4.0BSD under SIMH:
Install 4.3BSD on SIMH:
Marshall Kirk McKusick, George V. Neville-Neil, Robert N.M. Watson. The Design and Implementation of the FreeBSD Operating System, 2nd ed., Addison-Wesley Professional, 2014. ISBN-13: 978-0321968975, ISBN-10: 0321968972
McKusick, Marshall Kirk; Neville-Neil, George V. The Design and Implementation of the FreeBSD Operating System, 1st Edition, Addison-Wesley Professional,2004. ISBN-13: 9780201702453 ISBN-10: 0201702452
McKusick, Marshall Kirk; Bostic, Keith; Karels, Michael J.; Quarterman, John S. The Design and Implementation of the 4.4 BSD Operating System, Addison-Wesley,1996. ISBN-13: 9780201549799 ISBN-10: 0201549794
Negus, Christopher; Caen, Francois BSD UNIX Toolbox: 1000+ Commands for FreeBSD, OpenBSD and NetBSD, Wiley, 2008. ISBN-13: 9780470376034 ISBN-10: 0470376031
Lucas, Michael W. Absolute FreeBSD, 3rd Edition: The Complete Guide to FreeBSD, No Starch Press, 2018,ISBN-13: 9781593278922 ISBN-10: 1593278926
Approved Reading from the Computer Science Department of Carnegie Mellon
ZFS – An Introduction of the Implementation of ZFS by Dr Marshall Kirk McKusick