Subject: patterns Digest V97 #32
Resent-Date: 19 May 1997 08:00:03 -0000
Resent-From: mlist1@jerry.cs.uiuc.edu
Resent-CC: recipient list not shown: ;
From: patterns-request@cs.uiuc.edu
To: patterns@cs.uiuc.edu

------------------------------

Content-Type: text/plain

patterns Digest                         Volume 97 : Issue 32

Today's Topics:
  Revised and renamed pattern: Literat  [ Dwig@MONTEREY (Don Dwiggins) ]

Administrivia:
To be added or removed from the patterns list, send e-mail to
patterns-request@cs.uiuc.edu with "subscribe" or "unsubscribe"
in the SUBJECT.  Anything else gets read by a human.

There is an archive of patterns related material available by anonymous
ftp from st.cs.uiuc.edu in /pub/patterns.

There is also a WWW page for patterns located at
    http://st-www.cs.uiuc.edu/users/patterns/patterns.html

There is also a mailing list for talking about the theory of patterns
instead of talking about particular patterns.  You can subscribe by
sending e-mail to patterns-discussion-request with the SUBJECT (not body)
of "subscribe" (without the quotes).

------------------------------

>Date: Sun, 18 May 1997 13:59:21 -0700 (PDT)
>From: Dwig@MONTEREY (Don Dwiggins)
>To: patterns@cs.uiuc.edu
Subject: Revised and renamed pattern: Literate Development
Message-Id: <199705182059.NAA10511@italy.it.earthlink.net>

Here's the second version of the pattern "Document Early and Often".  I've
changed the name to Literate Development (by analogy to Knuth's Literate
Programming) because the earlier name led some people to think more in terms
of creating documentation than of a conceptual tool for development.

I've also sent an HTML formatted version of this to Ralph for posting on the
patterns web site.

Don Dwiggins                     "The truth will make you free,
SEI Information Technology        but first it will make you miserable"
ddwiggins@sei-it.com              -- Tom DeMarco

----------------------------------------------

Intent
------
Improve development quality and reduce rework by "thinking
on paper".

Context
-------
An individual's software development effort, starting with a
specification (vague or precise, sketchy or complete).  The
effort is large enough to require more than a single
development session, and complex enough that it's easy to
get lost in the details, or to lose your working context
between sessions.  There is pressure to deliver as soon as
possible while maintaining acceptable quality.  A secondary
goal is to make the work product accessible to those who
need to understand or maintain it.  Also, you have available
tools for quickly writing down your thoughts, in the form of
text, tables, drawings, etc.

Forces
------
- You want (or need) to develop as rapidly as possible,
  without losing control.
- You need to keep the focus of the effort across several
  development sessions.
- When you're working rapidly, issues, problems, and
  other thoughts arise that can't be dealt with immediately
  and should be kept track of, but there's no clear place to
  put them.
- You want to produce a product that is as defect-free as
  possible, both for the sake of the product and to save time
  by reducing rework.
- You want your efforts to be understood and appreciated
  by others.
- You want your colleagues to understand the problems you
  face, and to review your approach to solving them and the
  artifacts you create as solutions.
- Engineers hate to write more than is absolutely
  necessary, and want to avoid unnecessary overhead in their
  activities.
- Managers tend to deprecate documentation ("we can do
  that once we've met the delivery date"), and in fact any
  activity that appears to delay code production.  This is
  reinforced by common practice: writing anything but code is
  considered secondary work, even distracting from the primary
  goal.
- It's hard enough to create a single complex artifact in
  the necessary detail and completeness, let alone two.

Applicability
-------------
When the size and/or complexity of the effort are beyond
what you can keep in working memory over the required
duration of the effort.

When the development problems and issues aren't trivial or
obvious.

When there's a good chance that alternatives will need to be
explored, or that addressing later issues will require
revisiting earlier decisions.

When the work product will have to be integrated with
others, maintained, and improved, either by you at a later
date or by others.

When early and regular reviews are either mandated or highly
desirable to contain risk and/or catch and remove defects as
soon as possible after they arise.

Solution
--------
As you begin to think about the problem and consider
alternate solutions, begin a document to record your
thoughts.  Maintain the discipline of writing what you're
thinking.  Initially, don't worry about clarity,
organization, etc.  "Don't get it right, just get it writ."
The document should be textual at first; later, as
appropriate, include tables, diagrams, or other
representational forms.  (Rather than a single document, you
may find it useful to have several physical documents of
various kinds; for simplicity, I'll continue to say "the
document" to refer to the aggregate.)

Include sections for unresolved problems and issues, an
agenda of things to do, pointers to other documents or
related efforts, etc. -- whatever you need or want to keep
track of.  Let the document help keep your work focused and
organized.  Use it as the basis for status reports, and for
dumping and restoring your working memory when you have to
be away from the effort for a while.

Make sure that the document is easy to set up when you begin
work, close at hand while you work, and logically close to
the artifacts you're building (e.g., in the same directory
as the source files).  There should be minimal effort to
find and open the document, and to move between the document
and the artifacts.

When you complete writing down a small unit of work, take a
short break to "pull back" from it, then reread it.  Try to
rewrite it to make it simpler, clearer, and better
organized.  This should be the "innermost loop" of your
personal process.

Periodically, review and reorganize the overall document to
reflect your emerging vision and concept of the problem,
solution, and relations between them.  In particular, do
this when the document begins to feel unwieldy, rambling,
maybe even both redundant and inconsistent.  This is the
next level process loop.

When the document gets large, create a table of contents or
index (or hyperlinks) to help find things in it.
At appropriate times, get the document (or relevant parts of
it) reviewed by peers.  In preparation for this, clean it up
and organize it to make it more easily accessible by others.
This is the outermost loop.

Use the document as a tool at all levels, right down to
describing the algorithms of individual functions/methods
(snippets of code are perfectly acceptable, as long as they
serve the function of explaining and clarifying).  When
design-level changes need to be made, make them in the
document first to make sure you understand all the
implications.  Use the document changes to drive your work
in changing the code; this will help to ensure that you
don't leave anything out, and that you've thought everything
through before you start coding.

Consequences
------------
Initially, the early stages of the development process will
take longer (the overhead will be high at first until you
get comfortable with this way of working).  The less you're
used to writing your thoughts, the longer and harder the
learning curve will be.  To make it easier, don't record
more than the most essential things at first.

The discipline of recording your thoughts will in itself
clarify them and help expose hidden lapses, conflicts, etc.
The discipline of periodically reorganizing the document
will expose issues in the relations among components,
conflicting assumptions, oversights, etc.

You have a place to record concerns, issues, things to be
done later, and other things you don't want to lose.

Many defects will be reduced or caught relatively early by
the process of  "thinking out loud", periodic
reorganization, and keeping track of things that need
attention.  The consequent reduction in rework will repay
the investment in the document; the rewards will be greater
as you get better at the process.

When a peer review is in order, it will be relatively easy
to edit the document for easy comprehension by others, and
the document will facilitate a more thorough and effective
review.  This is a particularly important benefit in the
early stages of development, when reviews are the most
effective tools  to reduce risk and rework.

On completion, only a relatively quick editing and rewrite
process is needed to produce a good, accurate design
document and guide to the implementation.

The resulting work product will be much better prepared for
integration with others, testing, and maintenance work.

Implementation
--------------
Write first for yourself as audience (trick: as you're about
to write something, imagine you've just come back from a
month's vacation; what will this section/paragraph/sentence
need to contain to bring you back up to speed?).  Then write
for the next person who'll have to work with your product.
The major purpose of the document is to provide a form for
recording the evolving artifact(s) in a way that allows for
quick shifts of conceptual level and occasional convenient
reorganization; for this reason, the "matrix" of the
document should be English descriptions (or whatever
language you're most comfortable in). More formal languages
and notations can certainly be embedded in it where they
serve the purpose better than text.

When you feel like you've struggled to describe something
and it still isn't clear, step back and rethink it and the
context it occurs in.  You probably don't understand it well
enough to serve as a basis for further work on it.

When you're faced with two or more alternative ways to
proceed, and it's not clear which is best, record each of
the alternatives in its own section and work on each of them
a bit at a time, focusing on the salient differences among
them.  When a final choice is clear, condense the whole
thing into a summary of the alternatives considered and the
rationale for the choice.  You may have to revisit this
decision later.

You may occasionally get blocked: you can't think exactly
what to do next.  You have some possibilities, but none of
them seem very good.  Basic principle: "when you don't know
what to do, do something".  Take a break, then pursue one of
the possibilities, however dubious you are about it.  This
is usually enough to get you moving again.  The fact that
the doing something involves writing things down will be a
help here.

As design/implementation elements emerge, begin a unit test
plan as a section of the document.  If you're not sure how
you're going to test an element, rethink it.  This will help
ensure testability of the elements, and avoid leaving out
important tests.  Relate the test plan to the specifications
or requirements to ensure that you're not doing unnecessary
work, and that you're covering all requirements (and that
you understand each requirement well enough to be able to
devise a test for it). If QA people are available, have them
review the test plan when it's reasonably fleshed out.

When issues arise that need help from others, write them
first in the document, then extract them and use the extract
as the basis for interaction with the helpers.  Reflect
changes back into the document.  Mark the issues so they'll
be easy to find when you think you've covered them all, and
want to make sure.

As you begin creating artifacts, describe their logical and
physical organization in the document (directories, files,
images, repositories, ...).

When you're occasionally forced to work on the code first,
so it gets ahead of the document, be sure to go back and
update the document.  This is easiest to do if you don't let
the code get too far ahead.  At the very least, when you're
really under the gun, take a minute to mark the relevant
section(s) of the document as needing updating.

If possible, settle on and get familiar with a state-of-the-
art document editing product.  While it doesn't have to have
all the features of a publishing product, it should have at
least the following:
- Support for easily creating and reorganizing outlines.
- Support for easily making and reordering lists of things
  (numbered and bulleted).
- Support for "rich text".  While you won't need many
  fonts, it's useful to have a few highlighting and formatting
  styles available.  As the document grows, readability
  becomes important to assist ready comprehension and avoid
  forest/trees problems.
- Ability to easily exchange text with program editors.

The following additional capabilities are very handy.
- Support for "tables" (row-column layouts) in which you
  can type whole paragraphs in a cell.  (Not crucial, but very
  handy sometimes.)
- Ability to include diagrams, at least as pictures.
  When the document gets large, it'll help to have good
  organizing facilities like tables of contents, indexes, etc.

Related Patterns and Other Memes
- ------------------------------
Literate programming, initiated by Don Knuth, inspired the
title of this pattern, and shares some of the goals of this
pattern and some of its benefits.  The major difference is
that it focuses on the coding phase, and on producing
documentation as a byproduct.  For those unfamiliar with it,
here are some WWW references:
-    The seminal book by Knuth: http://www-cs-
  staff.stanford.edu/~knuth/lp.html.
-    A bibliography of works on the subject:
  http://bavi.unice.fr/Biblio/SE/litprog.html.
-    A good introduction to the concept:
http://www.ius.cs.cmu.edu/help/Programming/literate.html.

Andy Dent's "Diary Driven Development"
(http://www.highway1.com.au/adsoftware/sse.html, also
Software Development, April 1996). Very similar to this
pattern; Dent structures "the document" into a Site or
Project Diary, a Thoughts Diary, a Decisions Diary, and a
Code Change Diary.  Dent's emphasis is support for the "solo
software engineer", working on a one-person project.

Mercenary Analyst, one of Jim Coplien's process patterns
[Coplien].  The focus of that pattern is on documentation in
a team development environment, and recommends using an
experienced technical writer to create documentation for the
team.  One of the major problems a technical writer faces is
getting enough raw material to create the needed
documentation.  When the developers are practicing Literate
Development, this problem is largely alleviated, and the
writer can more easily weave the individual documents into a
good representation of the team effort.  Also, the writer
can assist developers in learning to record and organize
their thoughts.  Finally, a technologically knowledgeable
writer can serve as an "ongoing reviewer".

Write the User Manual First.  I don't think this has been
writtern as a pattern, but it's described well in the
original published book on the REXX language [Cowlishaw
1985], and I've heard of other uses.

Some of the points in the Implementation section could be
elaborated into patterns, e.g. Record Design Alternatives.

Examples of use
---------------
My own practice, for about the last 3 years.  Andy Dent's
diaries.  Private communications from others reporting
similar practices.

References
----------
[Coplien]
A Development Process Generative Pattern Language, in
Pattern Languages of Program Design, Addison-Wesley. 1995
and http://www.bell-labs.com/people/cope/Patterns/Process/index.html
[Cowlishaw 1985]
M. F. Cowlishaw, The REXX Language -- A Practical Approach
to Programming, Prentice Hall, Englewood Cliffs, NJ, 1985.

Acknowledgements
----------------
Many thanks to the people who responded to the previous
version, especially Brad Appleton, who has contributed
greatly to the changes in this version.

--------------------------------
End of patterns Digest V97 Issue #32
************************************
