Project

General

Profile

Development process » History » Version 19

Peter Amstutz, 09/25/2023 03:54 PM

1 1 Peter Amstutz
h1. Summary of Development Process
2
3
{{toc}}
4
5 8 Tom Clegg
h1. Issue descriptions
6
7
Consider starting with this outline (omitting empty/obvious sections):
8
* Background / context
9
* Current behavior
10
* Desired improvements
11
* Proposed implementation
12
* Exclusions / clarifications
13
* Open questions
14
15 1 Peter Amstutz
h1. Revision control
16
17
h2. Branches
18
19
* All development should be done in a branch.  The only exception to this should be trivial bug fixes.  What is trivial enough to not need review is the judgement of the developer, but when in doubt, ask for a review.
20
* Each story should be done in its own branch.
21
* Branch names are "####-story-summary" where #### is the redmine issue number followed by 3 or 4 words that summarize the story.
22
* Make your local branches track the main repository (@git push -u@)
23 9 Peter Amstutz
* Commit regularly, and push your branch to @git.arvados.org@ at the end of each day 
24
** Be paranoid, commits are cheap, pushing your commits to the remote repository is cheap, losing work is expensive
25 10 Peter Amstutz
** The preferred format of a commit message on a branch is like this (where 12345 should be replaced by the redmine issue number):
26
<pre>
27
12345: One line summary of changes in this commit
28
29
More detailed description of changes if relevant.
30
31
Arvados-DCO-1.1-Signed-off-by: Your Name <your.email@curii.com>
32
</pre>
33 1 Peter Amstutz
* Don't push uninvited changes to other developer's branches.
34
** To contribute to another developer's branch, check with them first, or create your own branch ("####-story-summary-ABC" where ABC are your initials) and ask the other developer to merge your branch.
35
36
h3. Merging
37
38 6 Ward Vandewege
Branches should not be merged to main until they are ready (see [[Summary of Development Process#Ready to merge|Ready to merge]] below).
39 1 Peter Amstutz
40
# @git remote -v@
41 6 Ward Vandewege
** Make sure your @origin@ is git.arvados.org, not github. *Don't push directly to the github main* branch -- let git.arvados.org decide whether it's OK to push to github.
42
# @git checkout main@
43 1 Peter Amstutz
# @git pull --ff-only@
44 6 Ward Vandewege
#* This ensures your main is up to date. Otherwise "git push" below might fail, and you'll be backtracking.
45 1 Peter Amstutz
# @git merge --no-ff branchname@
46
#* *The @--no-ff@ part is important!* It ensures there is actually a commit representing this merge. This is your opportunity to record the name of your branch being merged, and the relevant story number. Without it, the git history looks like we all just mysteriously started developing at the tip of your (now unnamed) feature branch.
47
#* In your merge commit message, *include the relevant story/issue number* (either "@refs #1234@" or "@closes #1234@").
48 4 Nico César
#* In your merge commit message, *include Arvados-DCO-1.1-Signed-off-by line* (i.e. Arvados-DCO-1.1-Signed-off-by: Jane Doe <jane@example.com>)
49 10 Peter Amstutz
#* The preferred format of a merge commit message is like this:
50
<pre>
51
Merge branch '12345-story-summary'
52
53
refs #12345
54
55
Arvados-DCO-1.1-Signed-off-by: Your Name <your.email@curii.com>
56
</pre>
57 1 Peter Amstutz
# @git push@
58 3 Peter Amstutz
# Look for Jenkins' build results at https://ci.arvados.org .
59 1 Peter Amstutz
60
h3. Rejected pushes
61
62 6 Ward Vandewege
We have a git hook in place that will reject pushes that do not follow these guidelines.  The goal of these policies is to ensure a clean linear history of changes to main with consistent cross referencing with issue numbers.  These policies apply to the commits listed on "git rev-list --first-parent" when pushing to main, and not to commits on any other branches.
63 1 Peter Amstutz
64
If you try to push a (set of) commit(s) that does not pass mustard, you will get a [POLICY] reject message on stdout, which will also list the offending commit. You can use
65
66
  git commit --amend
67
68
to update the commit message on your last commit, if that is the offending one, or else you can use 
69
70
  git rebase --interactive
71
72
to rebase and fix up a commit message on an earlier commit.
73 6 Ward Vandewege
74 1 Peter Amstutz
h4. All merge commits to main must be from a feature branch into main
75 6 Ward Vandewege
76 1 Peter Amstutz
Merges that go the other way (from main to a feature branch) that get pushed to main as a result of a fast-forward push will be rejected.  In other words:  when merging to main, make sure to use --no-ff.
77 6 Ward Vandewege
78 1 Peter Amstutz
h4. Merges between local and remote main branches will be rejected
79 6 Ward Vandewege
80 1 Peter Amstutz
Merges between local and remote main branches (generally merges created by "git pull") will be rejected, in order to maintain a linear main history.  If this happens, you'll need to reset main to the remote head and then remerge or rebase.
81
82
h4. Proper merge message format
83 6 Ward Vandewege
84 1 Peter Amstutz
All merge commits to main must include the text "Merge branch 'featurebranch'" or they will be rejected.
85
86 10 Peter Amstutz
h4. All commits to main include an issue number or explicitly say "no issue #"
87 1 Peter Amstutz
88 6 Ward Vandewege
All commits to main (both merges and single parent commits) must
89 11 Peter Amstutz
include the text "refs #", "closes #", "fixes #", or "no issue #" or they will be
90 1 Peter Amstutz
rejected.
91
92
h4. Avoid broken commit messages
93
94
Your commit message matches
95
96
  /Please enter a commit message to explain why this merge is necessary/
97
98
h2. Commit logs
99
100 5 Ward Vandewege
See https://dev.arvados.org/projects/arvados/wiki/Coding_Standards
101 1 Peter Amstutz
102
h2. Code review process
103
104
Code review has high priority! Branches shouldn't sit around for days waiting for review/merge.
105
106
When your branch is ready for review:
107
# Create/update a review task on the story so it looks like this:
108
#* subject = "review {branch name}"
109
#* state = in progress
110
#* assignee is not null
111 12 Tom Clegg
# Comment on the issue page (not the review page), including
112
#* branch name
113
#* commit hash
114
#* link to Jenkins test run
115
#* if appropriate, a brief description of what's in the branch (may be omitted if it's already in the commit messages, or if it would just be a copy of the issue subject/description)
116 1 Peter Amstutz
# Ping your reviewer (during daily standup, via e-mail and/or via chat).
117
118
Doing a review:
119 13 Peter Amstutz
# Reviewers are usually assigned at sprint kickoff, but if you don't have a reviewer, ask for a volunteer in chat and/or at daily stand-up.
120 1 Peter Amstutz
# When you start the review, assign the review task to yourself and move the review task to "in progress" to make sure other people don't duplicate your effort.
121 6 Ward Vandewege
# The recommended process for reviewing diffs for a branch is @git diff main...branchname@.  The reviewer must make sure that their repository is up to date (or use @git diff origin/main...origin/branchname@). Note the 3 dots (not two)
122 16 Peter Amstutz
# The reviewer goes through the [[#Ready-to-merge]] checklist
123 14 Peter Amstutz
# After doing a review, write up comments ("fix these problems" or "ready to merge") to the story page, make a note of the git commit revision that was reviewed, assign the review task back to the original developer, and notify the original developer by chat (or by some other means such as at daily standup).
124
#* In comments, it is helpful to indicate how strongly you feel how/important the comment is as "low", "medium", or "high"
125 7 Tom Clegg
#* low: nitpick not necessarily worth changing here if you don't feel like it, but I'm mentioning it to help improve habits
126
#* medium: suggestion/idea that you should at least acknowledge/respond to, even if we don't end up resolving it here
127 1 Peter Amstutz
#* high: we should make sure we both agree on how this is resolved before merging
128 7 Tom Clegg
# The original developer should address any outstanding problems/comments in the code, then write a brief response indicating which points were dealt with or intentionally rejected/not addressed.
129 14 Peter Amstutz
# If the response involves more commits, do that, then goto "branch is ready for review". This process iterates until the branch is deemed ready to merge (indicating by posting a comment with "LGTM" for "Looks Good To Merge")
130
# When the comments are all low priority, someone might write something like "LGTM if you fix this one typo", this indicates that once the minor comments are handled (fixed or responded to) that the branch should be merged without another review cycle.
131 1 Peter Amstutz
# Once the branch is merged, move the "review" task to "resolved".
132
133
To list unmerged branches:
134 6 Ward Vandewege
* Yours: @git branch --no-merged main@
135
* Everyone: @git branch -a --no-merged main@
136 1 Peter Amstutz
137
h2. Ready to merge
138
139 19 Peter Amstutz
See also the "Ready-to-merge-checklist":https://dev.arvados.org/projects/arvados/wiki/Coding_Standards#Ready-to-merge-checklist
140
141 1 Peter Amstutz
When merging, both the developer and the reviewer should be convinced that:
142 6 Ward Vandewege
* Current/recent main is merged. (Otherwise, you can't predict what merge will do.)
143 1 Peter Amstutz
* The branch is pushed to git.arvados.org
144
* The code is suitably robust.
145
* The code is suitably readable.
146
* The code is suitably scalable. For example, client code is not allowed to print or sort unbounded lists. If the code handles a list of items, consider what happens when the list is 10x as large as you expect. What about 100x? A million times?
147
* The code accomplishes what the story specified. If not, explain why (e.g., the branch is only part of the story, a better solution was found, etc.) in the issue comments
148
* New API names (methods, attributes, error codes) and behaviors are well chosen. It sucks to change them later, and have to choose between compatibility and greatness.
149
* Tests that used to pass still pass. (Be extremely careful when altering old tests to make them pass. Do not change existing tests to test new code. Add assertions and write new tests. If you change or remove an existing test, you are breaking behavior that someone already decided was worth testing!)
150
* Recent clients/SDKs work against the new API server. (Things rarely turn out well when we rely on all clients being updated at once in lockstep with the API server. Our test suite doesn't check this for us yet, so for now we have to pay attention.)
151
* New/fixed behavior is tested. (Although sometimes we decide not to block on inadequate testing infrastructure... that sucks!)
152
* New/changed behavior is documented. Search the doc site for relevant keywords to help you find the right sections.
153
* Whitespace errors are not committed. (Tab characters, spaces at EOL, etc.)
154 18 Peter Amstutz
* Git commit messages are descriptive. If they aren't, this is your last chance to rebase/reword.
155
* Code meets other [[arvados:Coding Standards]]
156
* For GUI work: user interface elements meet accessibility guidelines on the coding standards page
157 1 Peter Amstutz
158
h2. Handling pull requests from github
159
160
_This is only for contributions by *external contributors*, i.e., people who don't have permission to write directly to arvados.org repositories._
161
162 6 Ward Vandewege
First make sure your main is up to date.
163 1 Peter Amstutz
164 6 Ward Vandewege
    git checkout main; git pull --ff-only
165 1 Peter Amstutz
166
*Option 1:* On the pull request page on github, click the "You can also merge branches on the command line" link to get instructions.
167
168
* Don't forget to run tests.
169
170
*Option 2:* (a bit shorter)
171
172 6 Ward Vandewege
Say we have "chapmanb  wants to merge 1 commit into arvados:main from chapmanb:branchname"
173 1 Peter Amstutz
* @git fetch https://github.com/chapmanb/arvados.git branchname:chapmanb-branchname@
174
* @git merge --no-ff chapmanb-branchname@
175
* Use the commit message: @Merge branch 'branchname' from github.com/chapmanb. No issue #@
176
(or @refs #1234@ if there is an issue#)
177 6 Ward Vandewege
* Confirm diff: @git diff origin/main main@
178 1 Peter Amstutz
* Run tests
179
* @git push@
180
181
h1. Non-fast-forward push
182
183
Please don't get into a situation where this is needed.
184
185 6 Ward Vandewege
# On dev box: @git push -f git@github.com:arvados/arvados proper_head_commit:main proper_head_commit:staging@
186
# On dev box: @git push -f git@git.arvados.org:arvados.git proper_head_commit:main@
187
# As gitsync@dev.arvados.org: @cd /scm/arvados; git fetch origin; git checkout main; git reset --hard origin/main@
188 1 Peter Amstutz
189
(At least that's what TC did on 2016-03-10. We'll see how it goes.)
190
191
h1. Working with external upstream projects
192
193
Development process summary (1-6 should follow the guidelines above)
194
195
# Each feature is developed in a git branch named @<issue_number>-<summary>@, for example @12521-web-app-config@
196
# Each feature has a "Review" task.  You can see the features and review tasks on the task board.
197
# When the feature branch is ready for review, update the title of the Review task to say "Review <branchname>" and move it from the *New* column the to *In Progress* column
198
# The reviewer responds on the issue page with questions or comments
199
# When the branch is ready to merge, the reviewer will add a comment "Looks Good To Me" (LGTM) on the issue page
200
# Merge the feature into into the Arvados main branch
201
# Push the feature branch to github and make a pull request (PR) of the branch against the external project upstream
202
# Handle code review comments/change requests from the external project team
203 6 Ward Vandewege
# Once the external project merges the PR, merge external project upstream main back into the feature branch
204 1 Peter Amstutz
# Determine if external project upstream brings any unrelated changes that breaks things for us
205
# If necessary, make fixes, make a new PR, repeat until stable
206 6 Ward Vandewege
# Merge the feature branch (now up-to-date with external project upstream) into Arvados main
207 1 Peter Amstutz
208
This process is intended to let us work independently of how quickly the external project team merges our PRs, while still maximizing the chance that they will be able to accept our PRs by limiting the scope to one feature at a time.
209
210 6 Ward Vandewege
This assumes using git merge commits and avoiding rebases, so we can easily perform merges back and forth between the three branches (Arvados main, feature, external project main).
211 1 Peter Amstutz
212
213
h1. Scrum
214
215
h2. References
216
217
These books give us a reference point and vocabulary. 
218
219
* _Essential Scrum: A Practical Guide to the Most Popular Agile Process_ by Kenneth Rubin
220
* _User Stories Applied: For Agile Software Development_ by Mike Cohen
221
222
h2. Roles
223
224
225
h3. Product Owner
226
227
* Decide what goes on the backlog
228
* Decide backlog priorities
229
* Work with stakeholders to understand their requirements and priorities
230
* Encode stakeholder requirements/expectations as user stories
231
* Lead sprint planning meetings 
232
* Lead release planning meetings 
233
* Lead product planning meetings 
234
* Lead Sprint Kick-off Meetings
235
* Lead Sprint Review Meetings
236
* Decide on overall release schedule 
237
238
h3. Scrum Master
239
240
* Lead Daily Scrum Meeting
241
* Help to eliminate road blocks 
242
* Lead Sprint Retrospective Meetings
243
* Organize Sprint Schedule 
244
* Help team organize and stay on track with Scrum process
245
* Teach new engineers how Scrum works
246
247
h3. Top stakeholders
248
249
* Conduct market research 
250
* Synthesize market research into user stories 
251
* Work with Product Owner to prioritize stories
252
* Define overall business goals for product 
253
* Work with Product Owner to define overall release cycle 
254
* Organize User Input and dialog with users for engineering team 
255
* Contribute to backlog grooming 
256
* Bring voice of customer into planning process 
257
* Define user personas 
258
* Coordinate user communication 
259
* Develop technical marketing and sales materials 
260
* Assist sales team in presenting product value proposition
261
* Train sales in technical aspects of the product
262
263
h2. Definition of Done
264
265
An issue is resolved when:
266
267
* Code is written
268
* Existing code is refactored if appropriate
269
* Documentation is written/updated
270
* Acceptance tests are satisfied
271 6 Ward Vandewege
* Code is merged in main
272 1 Peter Amstutz
* All Jenkins jobs pass (test, build packages, deploy to dev clusters)
273
* Feature works on applicable dev clusters
274
275
h2. Standard Schedule
276
277
Sprints are two weeks long. They start and end on Wednesdays.
278
279
h3. Key meetings
280
281 2 Peter Amstutz
Every day:
282 1 Peter Amstutz
283
Daily Scrum (15 Minutes)
284
Who: Development team, product owner. Silent observers welcome.
285
* What did you do yesterday?
286
* What will you do today?
287
* What obstacles are in your way?
288
289
h4. Sprint review & kickoff (every 2 weeks on Wednesday):
290
291
Sprint Review (30 minutes)
292
Who: Development team, product owner, stakeholders.
293
* Demo of each feature built and relationship to stories
294
* Product owner explains which backlog items are done
295
* Development team demonstrates the work done, and answers questions about the sprint increment
296
* Product owner discusses the backlog as it stands. Revise expected completion dates based on recent progress (if needed)
297
* Review current product status in context of business goals
298
299
Sprint Retrospective (30 minutes)
300
Who: Development team, product owner.
301
* Review what processes worked well, and what didn't, in the sprint just finished
302
* Propose and agree to changes to improve future sprints
303
* Assign action items (meetings/tasks) to implement agreed-upon process improvements
304
305
Sprint Kick Off (1 hour)
306
Who: Development team, product owner.
307
* Add latest bugs or dependencies to sprint
308
* Create tasks for each story
309
* Assign a developer to each task
310
* Assign an on-call engineer for that sprint who will triage customer support requests
311
* Check that commitment level is realistic
312
313
h4. Planning (alternate Wednesdays mid-sprint)
314
315 2 Peter Amstutz
Roadmap review (1 hour)
316 1 Peter Amstutz
Who: Development team, product owner, stakeholders.
317 2 Peter Amstutz
* Report high level status of epics
318
* Prioritize epics
319
* Define new epics
320 1 Peter Amstutz
321 2 Peter Amstutz
Sprint Planning (1-2 hours)
322 1 Peter Amstutz
Who: Development team, product owner.
323 2 Peter Amstutz
* Discuss and get engineering team consensus on feature design & implementation strategy for tasks on current and upcoming epics