diff --git a/resources/tests/readability/folha/expected.html b/resources/tests/readability/folha/expected.html new file mode 100644 index 0000000..d67fefe --- /dev/null +++ b/resources/tests/readability/folha/expected.html @@ -0,0 +1,23 @@ +
+

+ Após rechaçar um encontro da seleção brasileira com o presidente eleito JairBolsonaro, o técnico Tite declarou que errou ao levar a taça da Copa Libertadores de 2012, conquistada pelo Corinthians, ao ex-presidente Luiz Inácio Lula da Silva. +

+

+ Ao lado de representantes do clube paulista, o atual comandante do Brasil ainda entregou uma réplica do troféu a Lula. +

+

+ "Em 2012 eu errei. Ele não era presidente, mas fui ao Instituto e mandei felicitações por um aniversário. Não me posicionei politicamente. Não tenho partido político, tenho sim a torcida para que o Brasil seja melhor em igualdade social. E que nossas prioridades sejam educação e punição. Que seja dada a possibilidade de estudo ao garoto de São Braz, que não tem chão batido para ir à escola, ou da periferia de Caixas ou do morro do Rio de Janeiro. Seja dada a ele a prioridade de estudo e não a outras situações", falou Tite ao programa "Grande Círculo", que ainda irá ao ar no SporTV. +

+

+ Na ocasião, Tite e outros representantes do Corinthians foram ao Instituto Lula para mostrar a taça original da Libertadores ao ex-presidente. +

+

+ O assunto foi levantado porque recentemente Tite foi questionado se aceitaria um encontro da seleção brasileira com Bolsonaro em uma conquista de título ou antes da Copa América de 2019, por exemplo. O treinador deixou claro que preferiria evitar esse tipo de formalidade. +

+

+ Apesar disso, Tite não questionou a ação de Palmeiras e CBF, que convidaram Bolsonaro para a festa do título do Campeonato Brasileiro. O presidente eleito até levantou a taça conquistada pelo clube alviverde. +

+

+ "Em 2012 eu fiz e errei. O protocolo e a situação gerada no jogo do Palmeiras são fatos de opinião pessoal. CBF e Palmeiras, enquanto instituições têm a opinião. Errei lá atrás, não faria com o presidente antes da Copa e nem agora porque entendo que misturar esporte e política não é legal. Fiz errado lá atrás? Sim. Faria de novo? Não", acrescentou o comandante. +

+
diff --git a/resources/tests/readability/folha/source.html b/resources/tests/readability/folha/source.html new file mode 100644 index 0000000..03f749c --- /dev/null +++ b/resources/tests/readability/folha/source.html @@ -0,0 +1,2518 @@ + + + + + + + + Tite diz que errou ao levar taça da Libertadores a Lula em 2012 - 21/12/2018 - Esporte - Folha + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + +
+
+ +
+ +
+
+
+
+ +
+
+
+ +
+
+
+
+
+ + + +
+ +
+ + + +
+
+ +
+ + + +
+
+
+
+
+
+
+
+
+

+ Tite diz que errou ao levar taça da Libertadores a Lula em 2012 +

+

+ Na ocasião, técnico do Corinthians entregou réplica do troféu ao ex-presidente +

+
+
+
+
    +
  • + +
  • +
  • +
    + + +
    +
  • +
+
+
+
+
+
+ Luiz Inácio Lula da Silva (ao centro) recebe de Tite e outros representantes do Corinthians réplica da taça da Libertadores +
+ Luiz Inácio Lula da Silva (ao centro) recebe de Tite e outros representantes do Corinthians réplica da taça da Libertadores - Heinrich Aikawa/Instituto Lula +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+
+
+ São Paulo +
+
+

+ Após rechaçar um encontro da seleção brasileira com o presidente eleito Jair Bolsonaro, o técnico Tite declarou que errou ao levar a taça da Copa Libertadores de 2012, conquistada pelo Corinthians, ao ex-presidente Luiz Inácio Lula da Silva. +

+

+ Ao lado de representantes do clube paulista, o atual comandante do Brasil ainda entregou uma réplica do troféu a Lula. +

+

+ "Em 2012 eu errei. Ele não era presidente, mas fui ao Instituto e mandei felicitações por um aniversário. Não me posicionei politicamente. Não tenho partido político, tenho sim a torcida para que o Brasil seja melhor em igualdade social. E que nossas prioridades sejam educação e punição. Que seja dada a possibilidade de estudo ao garoto de São Braz, que não tem chão batido para ir à escola, ou da periferia de Caixas ou do morro do Rio de Janeiro. Seja dada a ele a prioridade de estudo e não a outras situações", falou Tite ao programa "Grande Círculo", que ainda irá ao ar no SporTV. +

+

+ Na ocasião, Tite e outros representantes do Corinthians foram ao Instituto Lula para mostrar a taça original da Libertadores ao ex-presidente. +

+
+ +
+

+ O assunto foi levantado porque recentemente Tite foi questionado se aceitaria um encontro da seleção brasileira com Bolsonaro em uma conquista de título ou antes da Copa América de 2019, por exemplo. O treinador deixou claro que preferiria evitar esse tipo de formalidade. +

+

+ Apesar disso, Tite não questionou a ação de Palmeiras e CBF, que convidaram Bolsonaro para a festa do título do Campeonato Brasileiro. O presidente eleito até levantou a taça conquistada pelo clube alviverde. +

+

+ "Em 2012 eu fiz e errei. O protocolo e a situação gerada no jogo do Palmeiras são fatos de opinião pessoal. CBF e Palmeiras, enquanto instituições têm a opinião. Errei lá atrás, não faria com o presidente antes da Copa e nem agora porque entendo que misturar esporte e política não é legal. Fiz errado lá atrás? Sim. Faria de novo? Não", acrescentou o comandante. +

+
+
+ UOL +
+
+ +
+
+ Tópicos relacionados + +
+ + +
+
+

+ Comentários +

+

+ Os comentários não representam a opinião do jornal; a responsabilidade é do autor da mensagem. +

+ +
+ +
+ + +
+
+
+
+ +
+
+
+
+
+ +
+
+ +
+
+ +
+
+
+
+ +
+
+
+
+ +
+
+
+ NEWSLETTER +
+ +
+
+
+ + + + + + + + + + diff --git a/resources/tests/readability/gmw/expected.html b/resources/tests/readability/gmw/expected.html new file mode 100644 index 0000000..e06e352 --- /dev/null +++ b/resources/tests/readability/gmw/expected.html @@ -0,0 +1,43 @@ +
+

  翱翔于距地球数千公里的太空中,进入广袤漆黑的未知领域,是一项艰苦卓绝的工作。这让人感到巨大压力和极度恐慌。那么,为什么不能让宇航员来一杯“地球末日”鸡尾酒来放松一下?

+

  不幸的是,对于希望能喝上一杯的太空探险者,那些将他们送上太空的政府机构普遍禁止他们染指包括酒在内的含酒精饮料。

+

  但是,很快普通人都会有机会向人类“最终的边疆”出发——以平民化旅行的形式,去探索和殖民火星。确实,火星之旅将是一次令人感到痛苦的旅行,可能一去不复返并要几年时间才能完成,但是否应该允许参与者在旅程中痛饮一番?或至少携带能在火星上发酵自制酒精饮料的设备?

+

(Credit: Nasa)

+

  图注:巴兹?奥尔德林(Buzz Aldrin)可能是第二个在月球上行走的人,但他是第一个在月球上喝酒的人

+

  事实是,历史上酒与太空探险有一种复杂的关系。让我们来看看喝了酒的航天员究竟会发生什么—— 如果我们开始给予进入太空的人类更大的自由度,又可能会发生什么。

+

  人们普遍认为,当一个人所处的海拔越高,喝醉后会越容易感到头昏。因此,人们自然地想到,当人身处地球轨道上时,饮酒会对人体有更强烈的致眩作用。但这种说法可能不是正确的。

+

  事实上,有证据表明,早在上世纪八十年代就澄清了这一传言。1985年,美国联邦航空管理局(UFAA)开展了一项研究,以验证人在不同的海拔高度饮酒,是否会影响执行复杂任务时的表现和酒精测定仪的读数。

+

  在这项研究中,17名男子被要求在地面和一间模拟海拔3.7公里的房间内喝下一些伏特加。然后,他们被要求完成各种任务,包括心算口算问题、用操纵杆在示波器上跟踪灯光以及各种其它测试。研究人员发现,“酒精和海拔高度对酒精测定仪读数或完成任务的表现情况没有交互作用”。

+

  所以,人乘坐飞机时醉得更快是个传说?纽约州立大学(State University of New York,SUNY)社会学荣誉教授戴夫·汉森(Dave Hanson)研究酒精和饮酒超过40年,他认为确实如此。他说:“我不认为它(在太空中饮酒)会有任何不同。”

+

  他认为高原反应可能类似于宿醉,但它也可能类似于中毒。他说:“如果人们没有感受到充分的大气压力,他们也会觉得喝醉了一样。”

+

  相反,那些声称在飞机上比在地面上醉得更快的人,可能只是经历了“自认喝醉(think-drink)”效应,这种效应多年来已被广泛研究。它表明,如果人们认为自己喝醉了,那他们的一举一动会真的像喝醉了一样—— 而不是实际上他们真的醉了。

+

  汉森指出:“如果人们脑子里一直认为在飞机上酒精会对他们产生与平常不同的作用,那么他们乘坐飞机时真的会觉得酒精对他们产生了不同的作用。”

+

  所以,如果酒精对人体的物理效应与海拔高度无关,那么在国际空间站上睡前小饮一杯不应该是一个大问题,对吧?错了。

+

  美国宇航局约翰逊航天中心发言人丹尼尔·霍特(Daniel Huot)表示:“国际空间站上的宇航员不允许喝酒。在国际空间站上,酒精和其它挥发性化合物的使用受到控制,因为它们的挥发物可能对该站的水回收系统产生影响。”

+

  为此,国际空间站上的宇航员甚至没有被提供含有酒精的产品,例如漱口水、香水或须后水。如果在国际空间站上饮酒狂欢,溢出的啤酒也可能存在损坏设备的风险。

+

(Credit: iStock)

+

  图注:测试表明,有关人在高空中喝酒更容易醉的传言是不正确的

+

  然后是责任的问题。我们不允许汽车司机或飞机飞行员喝醉后驾驶,所以并不奇怪同样的规则适用于国际空间站上的宇航员。毕竟国际空间站的造价高达1500亿美元,而且在接近真空的太空中其运行速度达到了每小时27680公里。

+

  然而,2007年,美国宇航局(NASA)成立了一个负责调查宇航员健康状况的独立小组,称历史上该机构至少有两名宇航员在即将飞行前喝了大量的酒,但仍然被允许飞行。Nasa安全负责人随后的审查发现并没有证据支持这一指控。宇航员在飞行前12小时是严禁饮酒的,因为他们需要充分的思维能力和清醒的意识。

+

  出台这一规则的原因很清楚。在1985年UFAA开展的关于酒精在不同海拔高度影响的研究中,研究人员得出结论,酒精的影响与海拔高度无关。无论参与测试的人员在什么海拔高度喝酒,其酒精测量仪的读数都是一样的。他们的行为表现受到的影响也相同,但如果提供给测试人员的是安慰剂,则身处高空比身处海平面的行为表现要更差一些。这表明,无论是否摄入酒精,海拔高度可能对心理表现有轻微的影响。

+

  国际空间站禁止享用啤酒等有大量泡沫的饮料,可能有另一个原因:没有重力的帮助,液体和气体会在宇航员的胃里不停地翻滚,导致他们不断地打嗝。

+

  然而,尽管有严格的规则,这并不意味着太空中的人类不会接触发酵液体。在国际空间站上进行了大量有关酒精的实验—— 但没有发生让众人去饮酒的情况,所以没有人真正了解太空中人体对酒精具体有怎样的反应。

+

  NASA发言人斯蒂芬妮?席尔霍尔茨(Stephanie Schierhol)表示:“我们研究了太空中宇航员身体的各种变化,包括微生物层面的。我们有一个营养计划,以确保他们的身体获得保持健康所需要的营养。显然,在实施‘天空实验室(skylab)’项目时,他们曾将雪利酒与宇航员一起送到太空中,但宇航员在零重力飞行时使用雪利酒的测试结果不太好。”天空实验室是美国第一座空间站。

+

  席尔霍尔茨补充说,在测试中使用雪利酒“引发呕吐反射,公众也反对”。

+

  也许最令人惊讶的是,人类在月球表面上喝的第一种液体是葡萄酒。前NASA宇航员巴兹·奥尔德林(Buzz Aldrin)在采访和他撰写的书中表示,1969年,在和尼尔·阿姆斯特朗(Neil Armstrong)走出登月舱之前的圣餐仪式上,他喝了少量葡萄酒。举行这一仪式时与地面的通信出现了暂停,因此这一过程从来没有播出。

+

  虽然Nasa对太空中酒精的使用有严格的规定,但在这方面俄罗斯过去似乎更为宽松。在其“和平号”空间站上,宇航员允许喝点干邑和伏特加。当他们发现国际空间站将严格禁止饮酒时,显然有不少怨言。

+

  然而,奇怪的是,酒仍然能通过各种方式出现在国际空间站上。2015年,日本酿酒商三得利(Suntory)的全球创新中心将该公司一些获奖的威士忌运送到国际空间站,参与一项旨在验证“能否通过利用微重力环境增强酒精饮料醇厚性”的实验。换句话说,在微重力下酒的陈酿过程可能不同,导致它的陈酿进程更快、味道更好。对此,地球上的每家酿酒商都想进一步地了解。

+

  几年前,即2011年9月至2014年9月,Nasa赞助了一个试验,研究微重力环境对威士忌中未发酵麦芽与烧焦橡木颗粒的影响,这两种物质能对威士忌的陈酿起帮助作用。在太空中逗留将近1000天后,用于测试的威士忌的单宁成分保持不变——但是太空中橡木颗粒产生了更高浓度的木质素分解产物,这种物质能赋予威士忌特别的风味。

+

  Nasa表示:“这种试验不仅对麦芽威士忌行业有影响,而且对整个食品和饮料行业也有影响。送上太空的威士忌与对照样品之间的风味差异是如此显著,需要进一步分析以破解不同口味产生的原因。”

+

  因此,即使宇航员自己被禁止在地球轨道上饮酒,但他们正在做的工作可以提高在地上消费的酒的质量。

+

  相比之下,执行登陆火星任务的人将远离家乡几年,而不是几个月,因此可能会有人提出有关禁止饮酒的规定可以放松一些。

+

  然而,像戴夫?汉森这样的专家认为,继续禁止饮酒并没有什么害处。除了实际的安全问题,饮酒还可能有其它挑战。汉森认为,地球人存在许多社会文化方面的差异,而且人连续几年时间呆在一个狭小的空间里,很容易突然发怒,这些因素都使饮酒问题变得很棘手。

+

(Credit: David Frohman/Peachstate Historical Consulting Inc)

+

  图注:奥尔德林的圣餐杯回到了地球上

+

  他说:“这是一个政治问题,也是一个文化方面的问题,但不是一个科学上的问题。这将是未来一个可能产生冲突领域,因为人们具有不同的文化背景,他们对饮酒的态度不同。”他进一步指出,如果你与穆斯林、摩门教徒或禁酒主义者分配在同一间宿舍怎么办?面对未来人们可能在一个没有期限的时间内呆在一个有限的空间里,需要“尽早解决”如何协调不同文化观点的问题。

+

  所以,当宇航员在地球轨道上时,将还不得不满足于通过欣赏外面的景色来振作精神,而不要指望沉溺于烈酒中。我们留在地球上的人,则可以准备好适量的香槟酒,以迎接他们的归来。

+

  原标题:他晚于阿姆斯特朗登月 却是首个敢在月球喝酒的人

+

  出品︱网易科学人栏目组 胖胖

+

  作者︱春春

+

[责任编辑:肖春芳]

+
diff --git a/resources/tests/readability/gmw/source.html b/resources/tests/readability/gmw/source.html new file mode 100644 index 0000000..048d101 --- /dev/null +++ b/resources/tests/readability/gmw/source.html @@ -0,0 +1,2103 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 宇航员在太空中喝酒会怎么样?后果很严重 _探索者 _光明网 + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + +
+ +
+
+ +
+ +
+ + + + +
+
+
+ +
+ + + + + + + + + + +
+ +
+
+
+ 首页 + > 科技频道 + > 探索者 + > + 正文 +
+ +

宇航员在太空中喝酒会怎么样?后果很严重

+ +
+ 2017-03-10 09:58 来源:网易科学人  +
+
+ 2017-03-10 09:58:03来源:网易科学人作者:责任编辑:肖春芳 +
+
+ + + +

  翱翔于距地球数千公里的太空中,进入广袤漆黑的未知领域,是一项艰苦卓绝的工作。这让人感到巨大压力和极度恐慌。那么,为什么不能让宇航员来一杯“地球末日”鸡尾酒来放松一下?

+

  不幸的是,对于希望能喝上一杯的太空探险者,那些将他们送上太空的政府机构普遍禁止他们染指包括酒在内的含酒精饮料。

+

  但是,很快普通人都会有机会向人类“最终的边疆”出发——以平民化旅行的形式,去探索和殖民火星。确实,火星之旅将是一次令人感到痛苦的旅行,可能一去不复返并要几年时间才能完成,但是否应该允许参与者在旅程中痛饮一番?或至少携带能在火星上发酵自制酒精饮料的设备?

+

(Credit: Nasa)

+

+   图注:巴兹?奥尔德林(Buzz Aldrin)可能是第二个在月球上行走的人,但他是第一个在月球上喝酒的人 +

+

  事实是,历史上酒与太空探险有一种复杂的关系。让我们来看看喝了酒的航天员究竟会发生什么—— 如果我们开始给予进入太空的人类更大的自由度,又可能会发生什么。

+

  人们普遍认为,当一个人所处的海拔越高,喝醉后会越容易感到头昏。因此,人们自然地想到,当人身处地球轨道上时,饮酒会对人体有更强烈的致眩作用。但这种说法可能不是正确的。

+

  事实上,有证据表明,早在上世纪八十年代就澄清了这一传言。1985年,美国联邦航空管理局(UFAA)开展了一项研究,以验证人在不同的海拔高度饮酒,是否会影响执行复杂任务时的表现和酒精测定仪的读数。

+

  在这项研究中,17名男子被要求在地面和一间模拟海拔3.7公里的房间内喝下一些伏特加。然后,他们被要求完成各种任务,包括心算口算问题、用操纵杆在示波器上跟踪灯光以及各种其它测试。研究人员发现,“酒精和海拔高度对酒精测定仪读数或完成任务的表现情况没有交互作用”。

+

  所以,人乘坐飞机时醉得更快是个传说?纽约州立大学(State University of New York,SUNY)社会学荣誉教授戴夫·汉森(Dave Hanson)研究酒精和饮酒超过40年,他认为确实如此。他说:“我不认为它(在太空中饮酒)会有任何不同。”

+

  他认为高原反应可能类似于宿醉,但它也可能类似于中毒。他说:“如果人们没有感受到充分的大气压力,他们也会觉得喝醉了一样。”

+

  相反,那些声称在飞机上比在地面上醉得更快的人,可能只是经历了“自认喝醉(think-drink)”效应,这种效应多年来已被广泛研究。它表明,如果人们认为自己喝醉了,那他们的一举一动会真的像喝醉了一样—— 而不是实际上他们真的醉了。

+

  汉森指出:“如果人们脑子里一直认为在飞机上酒精会对他们产生与平常不同的作用,那么他们乘坐飞机时真的会觉得酒精对他们产生了不同的作用。”

+

  所以,如果酒精对人体的物理效应与海拔高度无关,那么在国际空间站上睡前小饮一杯不应该是一个大问题,对吧?错了。

+

  美国宇航局约翰逊航天中心发言人丹尼尔·霍特(Daniel Huot)表示:“国际空间站上的宇航员不允许喝酒。在国际空间站上,酒精和其它挥发性化合物的使用受到控制,因为它们的挥发物可能对该站的水回收系统产生影响。”

+

  为此,国际空间站上的宇航员甚至没有被提供含有酒精的产品,例如漱口水、香水或须后水。如果在国际空间站上饮酒狂欢,溢出的啤酒也可能存在损坏设备的风险。

+

(Credit: iStock)

+

+   图注:测试表明,有关人在高空中喝酒更容易醉的传言是不正确的 +

+

  然后是责任的问题。我们不允许汽车司机或飞机飞行员喝醉后驾驶,所以并不奇怪同样的规则适用于国际空间站上的宇航员。毕竟国际空间站的造价高达1500亿美元,而且在接近真空的太空中其运行速度达到了每小时27680公里。

+

  然而,2007年,美国宇航局(NASA)成立了一个负责调查宇航员健康状况的独立小组,称历史上该机构至少有两名宇航员在即将飞行前喝了大量的酒,但仍然被允许飞行。Nasa安全负责人随后的审查发现并没有证据支持这一指控。宇航员在飞行前12小时是严禁饮酒的,因为他们需要充分的思维能力和清醒的意识。

+

  出台这一规则的原因很清楚。在1985年UFAA开展的关于酒精在不同海拔高度影响的研究中,研究人员得出结论,酒精的影响与海拔高度无关。无论参与测试的人员在什么海拔高度喝酒,其酒精测量仪的读数都是一样的。他们的行为表现受到的影响也相同,但如果提供给测试人员的是安慰剂,则身处高空比身处海平面的行为表现要更差一些。这表明,无论是否摄入酒精,海拔高度可能对心理表现有轻微的影响。

+

  国际空间站禁止享用啤酒等有大量泡沫的饮料,可能有另一个原因:没有重力的帮助,液体和气体会在宇航员的胃里不停地翻滚,导致他们不断地打嗝。

+

  然而,尽管有严格的规则,这并不意味着太空中的人类不会接触发酵液体。在国际空间站上进行了大量有关酒精的实验—— 但没有发生让众人去饮酒的情况,所以没有人真正了解太空中人体对酒精具体有怎样的反应。

+

  NASA发言人斯蒂芬妮?席尔霍尔茨(Stephanie Schierhol)表示:“我们研究了太空中宇航员身体的各种变化,包括微生物层面的。我们有一个营养计划,以确保他们的身体获得保持健康所需要的营养。显然,在实施‘天空实验室(skylab)’项目时,他们曾将雪利酒与宇航员一起送到太空中,但宇航员在零重力飞行时使用雪利酒的测试结果不太好。”天空实验室是美国第一座空间站。

+

  席尔霍尔茨补充说,在测试中使用雪利酒“引发呕吐反射,公众也反对”。

+

  也许最令人惊讶的是,人类在月球表面上喝的第一种液体是葡萄酒。前NASA宇航员巴兹·奥尔德林(Buzz Aldrin)在采访和他撰写的书中表示,1969年,在和尼尔·阿姆斯特朗(Neil Armstrong)走出登月舱之前的圣餐仪式上,他喝了少量葡萄酒。举行这一仪式时与地面的通信出现了暂停,因此这一过程从来没有播出。

+

  虽然Nasa对太空中酒精的使用有严格的规定,但在这方面俄罗斯过去似乎更为宽松。在其“和平号”空间站上,宇航员允许喝点干邑和伏特加。当他们发现国际空间站将严格禁止饮酒时,显然有不少怨言。

+

  然而,奇怪的是,酒仍然能通过各种方式出现在国际空间站上。2015年,日本酿酒商三得利(Suntory)的全球创新中心将该公司一些获奖的威士忌运送到国际空间站,参与一项旨在验证“能否通过利用微重力环境增强酒精饮料醇厚性”的实验。换句话说,在微重力下酒的陈酿过程可能不同,导致它的陈酿进程更快、味道更好。对此,地球上的每家酿酒商都想进一步地了解。

+

  几年前,即2011年9月至2014年9月,Nasa赞助了一个试验,研究微重力环境对威士忌中未发酵麦芽与烧焦橡木颗粒的影响,这两种物质能对威士忌的陈酿起帮助作用。在太空中逗留将近1000天后,用于测试的威士忌的单宁成分保持不变——但是太空中橡木颗粒产生了更高浓度的木质素分解产物,这种物质能赋予威士忌特别的风味。

+

  Nasa表示:“这种试验不仅对麦芽威士忌行业有影响,而且对整个食品和饮料行业也有影响。送上太空的威士忌与对照样品之间的风味差异是如此显著,需要进一步分析以破解不同口味产生的原因。”

+

  因此,即使宇航员自己被禁止在地球轨道上饮酒,但他们正在做的工作可以提高在地上消费的酒的质量。

+

  相比之下,执行登陆火星任务的人将远离家乡几年,而不是几个月,因此可能会有人提出有关禁止饮酒的规定可以放松一些。

+

  然而,像戴夫?汉森这样的专家认为,继续禁止饮酒并没有什么害处。除了实际的安全问题,饮酒还可能有其它挑战。汉森认为,地球人存在许多社会文化方面的差异,而且人连续几年时间呆在一个狭小的空间里,很容易突然发怒,这些因素都使饮酒问题变得很棘手。

+

(Credit: David Frohman/Peachstate Historical Consulting Inc)

+

+   图注:奥尔德林的圣餐杯回到了地球上 +

+

  他说:“这是一个政治问题,也是一个文化方面的问题,但不是一个科学上的问题。这将是未来一个可能产生冲突领域,因为人们具有不同的文化背景,他们对饮酒的态度不同。”他进一步指出,如果你与穆斯林、摩门教徒或禁酒主义者分配在同一间宿舍怎么办?面对未来人们可能在一个没有期限的时间内呆在一个有限的空间里,需要“尽早解决”如何协调不同文化观点的问题。

+

  所以,当宇航员在地球轨道上时,将还不得不满足于通过欣赏外面的景色来振作精神,而不要指望沉溺于烈酒中。我们留在地球上的人,则可以准备好适量的香槟酒,以迎接他们的归来。

+

  原标题:他晚于阿姆斯特朗登月 却是首个敢在月球喝酒的人

+

  出品︱网易科学人栏目组 胖胖

+

  作者︱春春 + +

+ +
[责任编辑:肖春芳]
+ +
+
+ +
+ +
+ + +
+
+
+ + + + + + + + + +
+ + + + + + + + +
+ + + + +
+ + + +
+ +
+
+
+
+
+
分享
+
+
+
+
+
+
0
+
+
+ +
+
+
+ + + + + + +
+ + +
+
+
您此时的心情
+
新闻表情排行 /
+
+
+ +
+
+ + + + +
+ + + + + + + + + +
+ + + + + +
+
+
+ +
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + + + + +
+
+
+
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + + + + +
+
+
+
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + +
+
+
+ + + + + +
+
+
+
+ + + + +
+ + + + + +
+
+

[值班总编推荐] + 想想童乞的遭遇,谁都会坐立不安
+

+

[值班总编推荐] + 网络慈善:珍视每一份爱心
+

+

[值班总编推荐] + 被村上春树打脸,日本右翼急了
+

+
+
+ +
+
+ + +
+ + + + + +
+
+ + + + + + +
+
+
+ + +
+ + + + + +
+
+ + + + + +
+
+ + + + + + + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + +
+ + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+
+ + +
+

手机光明网

+

光明网版权所有

+
+
+ +
+
+
+

光明日报社概况 | 关于光明网 | 报网动态 | 联系我们 | 法律声明 | 光明员工 | 光明网邮箱 | 网站地图

+

光明网版权所有

+
+ + + + + + + + + + + + +
+ + +
+ +
+ 立即打开
+ +
+ + + + + + + + + + + + + + + + +
+ + + + + + + + +
+ + +
+
X
+
+
选择其他平台 >>
分享到
+
+ +
+
+
+ +
+ + + diff --git a/resources/tests/readability/google-sre-book-1/expected.html b/resources/tests/readability/google-sre-book-1/expected.html new file mode 100644 index 0000000..8ec1d3e --- /dev/null +++ b/resources/tests/readability/google-sre-book-1/expected.html @@ -0,0 +1,302 @@ +

+ Monitoring Distributed Systems +

+

+ Google’s SRE teams have some basic principles and best practices for building successful monitoring and alerting systems. This chapter offers guidelines for what issues should interrupt a human via a page, and how to deal with issues that aren’t serious enough to trigger a page. +

+

+ Definitions +

+

There’s no uniformly shared vocabulary for discussing all topics related to monitoring. Even within Google, usage of the following terms varies, but the most common interpretations are listed here. +

+
+

+ Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. +

+

Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics. +

+

Testing externally visible behavior as a user would see it. +

+

An application (usually web-based) that provides a summary view of a service’s core metrics. A dashboard may have filters, selectors, and so on, but is prebuilt to expose the metrics most important to its users. The dashboard might also display team information such as ticket queue length, a list of high-priority bugs, the current on-call engineer for a given area of responsibility, or recent pushes. +

+

A notification intended to be read by a human and that is pushed to a system such as a bug or ticket queue, an email alias, or a pager. Respectively, these alerts are classified as tickets, email alerts,22 and pages. +

+

A defect in a software or human system that, if repaired, instills confidence that this event won’t happen again in the same way. A given incident might have multiple root causes: for example, perhaps it was caused by a combination of insufficient process automation, software that crashed on bogus input, and insufficient testing of the script used to generate the configuration. Each of these factors might stand alone as a root cause, and each should be repaired. +

+
+ Node and machine +
+
+

Used interchangeably to indicate a single instance of a running kernel in either a physical server, virtual machine, or container. There might be multiple services worth monitoring on a single machine. The services may either be: +

+
    +
  • Related to each other: for example, a caching server and a web server +
  • +
  • Unrelated services sharing hardware: for example, a code repository and a master for a configuration system like Puppet or Chef +
  • +
+
+

+ Any change to a service’s running software or its configuration. +

+

+ Why Monitor? +

+

There are many reasons to monitor a system, including: +

+
+

+ How big is my database and how fast is it growing? How quickly is my daily-active user count growing? +

+

+ Are queries faster with Acme Bucket of Bytes 2.72 versus Ajax DB 3.14? How much better is my memcache hit rate with an extra node? Is my site slower than it was last week? +

+

+ Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon. +

+

Dashboards should answer basic questions about your service, and normally include some form of the four golden signals (discussed in The Four Golden Signals). +

+

+ Our latency just shot up; what else happened around the same time? +

+
+

+ System monitoring is also helpful in supplying raw input into business analytics and in facilitating analysis of security breaches. Because this book focuses on the engineering domains in which SRE has particular expertise, we won’t discuss these applications of monitoring here. +

+

+ Monitoring and alerting enables a system to tell us when it’s broken, or perhaps to tell us what’s about to break. When the system isn’t able to automatically fix itself, we want a human to investigate the alert, determine if there’s a real problem at hand, mitigate the problem, and determine the root cause of the problem. Unless you’re performing security auditing on very narrowly scoped components of a system, you should never trigger an alert simply because "something seems a bit weird." +

+

+ Paging a human is a quite expensive use of an employee’s time. If an employee is at work, a page interrupts their workflow. If the employee is at home, a page interrupts their personal time, and perhaps even their sleep. When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a "real" page that’s masked by the noise. Outages can be prolonged because other noise interferes with a rapid diagnosis and fix. Effective alerting systems have good signal and very low noise. +

+ Setting Reasonable Expectations for Monitoring +

+

Monitoring a complex application is a significant engineering endeavor in and of itself. Even with substantial existing infrastructure for instrumentation, collection, display, and alerting in place, a Google SRE team with 10–12 members typically has one or sometimes two members whose primary assignment is to build and maintain monitoring systems for their service. This number has decreased over time as we generalize and centralize common monitoring infrastructure, but every SRE team typically has at least one “monitoring person.” (That being said, while it can be fun to have access to traffic graph dashboards and the like, SRE teams carefully avoid any situation that requires someone to “stare at a screen to watch for problems.”) +

+

In general, Google has trended toward simpler and faster monitoring systems, with better tools for post hoc analysis. We avoid "magic" systems that try to learn thresholds or automatically detect causality. Rules that detect unexpected changes in end-user request rates are one counterexample; while these rules are still kept as simple as possible, they give a very quick detection of a very simple, specific, severe anomaly. Other uses of monitoring data such as capacity planning and traffic prediction can tolerate more fragility, and thus, more complexity. Observational experiments conducted over a very long time horizon (months or years) with a low sampling rate (hours or days) can also often tolerate more fragility because occasional missed samples won’t hide a long-running trend. +

+

Google SRE has experienced only limited success with complex dependency hierarchies. We seldom use rules such as, "If I know the database is slow, alert for a slow database; otherwise, alert for the website being generally slow." Dependency-reliant rules usually pertain to very stable parts of our system, such as our system for draining user traffic away from a datacenter. For example, "If a datacenter is drained, then don’t alert me on its latency" is one common datacenter alerting rule. Few teams at Google maintain complex dependency hierarchies because our infrastructure has a steady rate of continuous refactoring. +

+

+ Some of the ideas described in this chapter are still aspirational: there is always room to move more rapidly from symptom to root cause(s), especially in ever-changing systems. So while this chapter sets out some goals for monitoring systems, and some ways to achieve these goals, it’s important that monitoring systems—especially the critical path from the onset of a production problem, through a page to a human, through basic triage and deep debugging—be kept simple and comprehensible by everyone on the team. +

+

+ Similarly, to keep noise low and signal high, the elements of your monitoring system that direct to a pager need to be very simple and robust. Rules that generate alerts for humans should be simple to understand and represent a clear failure. +

+ Symptoms Versus Causes +

+

Your monitoring system should address two questions: what’s broken, and why? +

+

+ The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. Table 6-1 lists some hypothetical symptoms and corresponding causes. +

+ + + + + + + + + + + + + + + + + + + + + + + + +
+Table 6-1. Example symptoms and causes +
SymptomCause

I’m serving HTTP 500s or 404s

+ Database servers are refusing connections +

My responses are slow

+ CPUs are overloaded by a bogosort, or an Ethernet cable is crimped under a rack, visible as partial packet loss +

Users in Antarctica aren’t receiving animated cat GIFs

+ Your Content Distribution Network hates scientists and felines, and thus blacklisted some client IPs +

Private content is world-readable

+ A new software push caused ACLs to be forgotten and allowed all requests +

+

+ "What" versus "why" is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise. +

+ Black-Box Versus White-Box +

+

We combine heavy use of white-box monitoring with modest but critical uses of black-box monitoring. The simplest way to think about black-box monitoring versus white-box monitoring is that black-box monitoring is symptom-oriented and represents active—not predicted—problems: "The system isn’t working correctly, right now." White-box monitoring depends on the ability to inspect the innards of the system, such as logs or HTTP endpoints, with instrumentation. White-box monitoring therefore allows detection of imminent problems, failures masked by retries, and so forth. +

+

+ Note that in a multilayered system, one person’s symptom is another person’s cause. For example, suppose that a database’s performance is slow. Slow database reads are a symptom for the database SRE who detects them. However, for the frontend SRE observing a slow website, the same slow database reads are a cause. Therefore, white-box monitoring is sometimes symptom-oriented, and sometimes cause-oriented, depending on just how informative your white-box is. +

+

+ When collecting telemetry for debugging, white-box monitoring is essential. If web servers seem slow on database-heavy requests, you need to know both how fast the web server perceives the database to be, and how fast the database believes itself to be. Otherwise, you can’t distinguish an actually slow database server from a network problem between your web server and your database. +

+

+ For paging, black-box monitoring has the key benefit of forcing discipline to only nag a human when a problem is both already ongoing and contributing to real symptoms. On the other hand, for not-yet-occurring but imminent problems, black-box monitoring is fairly useless. +

+ The Four Golden Signals +

+

The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four. +

+
+

The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors. +

+

A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per second. +

+

The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content. +

+
+

How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. +

+

+ In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. +

+

+ Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours." +

+
+
+

+ If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring. +

+ Worrying About Your Tail (or, Instrumentation and Performance) +

+

When building a monitoring system from scratch, it’s tempting to design a system based upon the mean of some quantity: the mean latency, the mean CPU usage of your nodes, or the mean fullness of your databases. The danger presented by the latter two cases is obvious: CPUs and databases can easily be utilized in a very imbalanced way. The same holds for latency. If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.23 If your users depend on several such web services to render their page, the 99th percentile of one backend can easily become the median response of your frontend. +

+

+ The simplest way to differentiate between a slow average and a very slow "tail" of requests is to collect request counts bucketed by latencies (suitable for rendering a histogram), rather than actual latencies: how many requests did I serve that took between 0 ms and 10 ms, between 10 ms and 30 ms, between 30 ms and 100 ms, between 100 ms and 300 ms, and so on? Distributing the histogram boundaries approximately exponentially (in this case by factors of roughly 3) is often an easy way to visualize the distribution of your requests. +

+ Choosing an Appropriate Resolution for Measurements +

+

Different aspects of a system should be measured with different levels of granularity. For example: +

+
    +
  • Observing CPU load over the time span of a minute won’t reveal even quite long-lived spikes that drive high tail latencies. +
  • +
  • On the other hand, for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime), probing for a 200 (success) status more than once or twice a minute is probably unnecessarily frequent. +
  • +
  • Similarly, checking hard drive fullness for a service targeting 99.9% availability more than once every 1–2 minutes is probably unnecessary. +
  • +
+

+ Take care in how you structure the granularity of your measurements. Collecting per-second measurements of CPU load might yield interesting data, but such frequent measurements may be very expensive to collect, store, and analyze. If your monitoring goal calls for high resolution but doesn’t require extremely low latency, you can reduce these costs by performing internal sampling on the server, then configuring an external system to collect and aggregate that distribution over time or across servers. You might: +

+
    +
  1. Record the current CPU utilization each second. +
  2. +
  3. Using buckets of 5% granularity, increment the appropriate CPU utilization bucket each second. +
  4. +
  5. Aggregate those values every minute. +
  6. +
+

+ This strategy allows you to observe brief CPU hotspots without incurring very high cost due to collection and retention. +

+ As Simple as Possible, No Simpler +

+

Piling all these requirements on top of each other can add up to a very complex monitoring system—your system might end up with the following levels of complexity: +

+
    +
  • Alerts on different latency thresholds, at different percentiles, on all kinds of different metrics +
  • +
  • Extra code to detect and expose possible causes +
  • +
  • Associated dashboards for each of these possible causes +
  • +
+

+ The sources of potential complexity are never-ending. Like all software systems, monitoring can become so complex that it’s fragile, complicated to change, and a maintenance burden. +

+

+ Therefore, design your monitoring system with an eye toward simplicity. In choosing what to monitor, keep the following guidelines in mind: +

+
    +
  • The rules that catch real incidents most often should be as simple, predictable, and reliable as possible. +
  • +
  • Data collection, aggregation, and alerting configuration that is rarely exercised (e.g., less than once a quarter for some SRE teams) should be up for removal. +
  • +
  • Signals that are collected, but not exposed in any prebaked dashboard nor used by any alert, are candidates for removal. +
  • +
+

+ In Google’s experience, basic collection and aggregation of metrics, paired with alerting and dashboards, has worked well as a relatively standalone system. (In fact Google’s monitoring system is broken up into several binaries, but typically people learn about all aspects of these binaries.) It can be tempting to combine monitoring with other aspects of inspecting complex systems, such as detailed system profiling, single-process debugging, tracking details about exceptions or crashes, load testing, log collection and analysis, or traffic inspection. While most of these subjects share commonalities with basic monitoring, blending together too many results in overly complex and fragile systems. As in many other aspects of software engineering, maintaining distinct systems with clear, simple, loosely coupled points of integration is a better strategy (for example, using web APIs for pulling summary data in a format that can remain constant over an extended period of time). +

+ Tying These Principles Together +

+

+ The principles discussed in this chapter can be tied together into a philosophy on monitoring and alerting that’s widely endorsed and followed within Google SRE teams. While this monitoring philosophy is a bit aspirational, it’s a good starting point for writing or reviewing a new alert, and it can help your organization ask the right questions, regardless of the size of your organization or the complexity of your service or system. +

+

When creating rules for monitoring and alerting, asking the following questions can help you avoid false positives and pager burnout:24

+
    +
  • Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?25 +
  • +
  • Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario? +
  • +
  • Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out? +
  • +
  • Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround? +
  • +
  • Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary? +
  • +
+

These questions reflect a fundamental philosophy on pages and pagers: +

+
    +
  • Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued. +
  • +
  • Every page should be actionable. +
  • +
  • Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page. +
  • +
  • Pages should be about a novel problem or an event that hasn’t been seen before. +
  • +
+

+ Such a perspective dissipates certain distinctions: if a page satisfies the preceding four bullets, it’s irrelevant whether the page is triggered by white-box or black-box monitoring. This perspective also amplifies certain distinctions: it’s better to spend much more effort on catching symptoms than causes; when it comes to causes, only worry about very definite, very imminent causes. +

+ Monitoring for the Long Term +

+

In modern production systems, monitoring systems track an ever-evolving system with changing software architecture, load characteristics, and performance targets. An alert that’s currently exceptionally rare and hard to automate might become frequent, perhaps even meriting a hacked-together script to resolve it. At this point, someone should find and eliminate the root causes of the problem; if such resolution isn’t possible, the alert response deserves to be fully automated. +

+

+ It’s important that decisions about monitoring be made with long-term goals in mind. Every page that happens today distracts a human from improving the system for tomorrow, so there is often a case for taking a short-term hit to availability or performance in order to improve the long-term outlook for the system. Let’s take a look at two case studies that illustrate this trade-off. +

+

Google’s internal infrastructure is typically offered and measured against a service level objective (SLO; see Service Level Objectives). Many years ago, the Bigtable service’s SLO was based on a synthetic well-behaved client’s mean performance. Because of problems in Bigtable and lower layers of the storage stack, the mean performance was driven by a "large" tail: the worst 5% of requests were often significantly slower than the rest. +

+

+ Email alerts were triggered as the SLO approached, and paging alerts were triggered when the SLO was exceeded. Both types of alerts were firing voluminously, consuming unacceptable amounts of engineering time: the team spent significant amounts of time triaging the alerts to find the few that were really actionable, and we often missed the problems that actually affected users, because so few of them did. Many of the pages were non-urgent, due to well-understood problems in the infrastructure, and had either rote responses or received no response. +

+

+ To remedy the situation, the team used a three-pronged approach: while making great efforts to improve the performance of Bigtable, we also temporarily dialed back our SLO target, using the 75th percentile request latency. We also disabled email alerts, as there were so many that spending time diagnosing them was infeasible. +

+

+ This strategy gave us enough breathing room to actually fix the longer-term problems in Bigtable and the lower layers of the storage stack, rather than constantly fixing tactical problems. On-call engineers could actually accomplish work when they weren’t being kept up by pages at all hours. Ultimately, temporarily backing off on our alerts allowed us to make faster progress toward a better service. +

In the very early days of Gmail, the service was built on a retrofitted distributed process management system called Workqueue, which was originally created for batch processing of pieces of the search index. Workqueue was "adapted" to long-lived processes and subsequently applied to Gmail, but certain bugs in the relatively opaque codebase in the scheduler proved hard to beat. +

+

+ At that time, the Gmail monitoring was structured such that alerts fired when individual tasks were “de-scheduled” by Workqueue. This setup was less than ideal because even at that time, Gmail had many, many thousands of tasks, each task representing a fraction of a percent of our users. We cared deeply about providing a good user experience for Gmail users, but such an alerting setup was unmaintainable. +

+

+ To address this problem, Gmail SRE built a tool that helped “poke” the scheduler in just the right way to minimize impact to users. The team had several discussions about whether or not we should simply automate the entire loop from detecting the problem to nudging the rescheduler, until a better long-term solution was achieved, but some worried this kind of workaround would delay a real fix. +

+

+ This kind of tension is common within a team, and often reflects an underlying mistrust of the team’s self-discipline: while some team members want to implement a “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it’s easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes. Managers and technical leaders play a key role in implementing true, long-term fixes by supporting and prioritizing potentially time-consuming long-term fixes even when the initial “pain” of paging subsides. +

+

+ Pages with rote, algorithmic responses should be a red flag. Unwillingness on the part of your team to automate such pages implies that the team lacks confidence that they can clean up their technical debt. This is a major problem worth escalating.

A common theme connects the previous examples of Bigtable and Gmail: a tension between short-term and long-term availability. Often, sheer force of effort can help a rickety system achieve high availability, but this path is usually short-lived and fraught with burnout and dependence on a small number of heroic team members. Taking a controlled, short-term decrease in availability is often a painful, but strategic trade for the long-run stability of the system. It’s important not to think of every page as an event in isolation, but to consider whether the overall level of paging leads toward a healthy, appropriately available system with a healthy, viable team and long-term outlook. We review statistics about page frequency (usually expressed as incidents per shift, where an incident might be composed of a few related pages) in quarterly reports with management, ensuring that decision makers are kept up to date on the pager load and overall health of their teams. +

+ Conclusion +

+

+ A healthy monitoring and alerting pipeline is simple and easy to reason about. It focuses primarily on symptoms for paging, reserving cause-oriented heuristics to serve as aids to debugging problems. Monitoring symptoms is easier the further "up" your stack you monitor, though monitoring saturation and performance of subsystems such as databases often must be performed directly on the subsystem itself. Email alerts are of very limited value and tend to easily become overrun with noise; instead, you should favor a dashboard that monitors all ongoing subcritical problems for the sort of information that typically ends up in email alerts. A dashboard might also be paired with a log, in order to analyze historical correlations. +

+

+ Over the long haul, achieving a successful on-call rotation and product includes choosing to alert on symptoms or imminent real problems, adapting your targets to goals that are actually achievable, and making sure that your monitoring supports rapid diagnosis. +

diff --git a/resources/tests/readability/google-sre-book-1/source.html b/resources/tests/readability/google-sre-book-1/source.html new file mode 100644 index 0000000..6fb2024 --- /dev/null +++ b/resources/tests/readability/google-sre-book-1/source.html @@ -0,0 +1,742 @@ + + + + + + + Google - Site Reliability Engineering + + + + + + + + + + + + +
+
+ +
+ +

+ Chapter 6 - Monitoring Distributed Systems +

+
+
+ + +
+
+
+
+
+
+

+ Monitoring Distributed Systems +

+ +

+ Google’s SRE teams have some basic principles and best practices for building successful monitoring and alerting systems. This chapter offers guidelines for what issues should interrupt a human via a page, and how to deal with issues that aren’t serious enough to trigger a page. +

+
+

+ Definitions +

+

+ There’s no uniformly shared vocabulary for discussing all topics related to monitoring. Even within Google, usage of the following terms varies, but the most common interpretations are listed here. +

+
+
+ Monitoring +
+
+

+ Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. +

+
+
+ White-box monitoring +
+
+

+ Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics. +

+
+
+ Black-box monitoring +
+
+

+ Testing externally visible behavior as a user would see it. +

+
+
+ Dashboard +
+
+

+ An application (usually web-based) that provides a summary view of a service’s core metrics. A dashboard may have filters, selectors, and so on, but is prebuilt to expose the metrics most important to its users. The dashboard might also display team information such as ticket queue length, a list of high-priority bugs, the current on-call engineer for a given area of responsibility, or recent pushes. +

+
+
+ Alert +
+
+

+ A notification intended to be read by a human and that is pushed to a system such as a bug or ticket queue, an email alias, or a pager. Respectively, these alerts are classified as tickets, email alerts,22 and pages. +

+
+
+ Root cause +
+
+

+ A defect in a software or human system that, if repaired, instills confidence that this event won’t happen again in the same way. A given incident might have multiple root causes: for example, perhaps it was caused by a combination of insufficient process automation, software that crashed on bogus input, and insufficient testing of the script used to generate the configuration. Each of these factors might stand alone as a root cause, and each should be repaired. +

+
+
+ Node and machine +
+
+

+ Used interchangeably to indicate a single instance of a running kernel in either a physical server, virtual machine, or container. There might be multiple services worth monitoring on a single machine. The services may either be: +

+
    +
  • Related to each other: for example, a caching server and a web server +
  • +
  • Unrelated services sharing hardware: for example, a code repository and a master for a configuration system like Puppet or Chef +
  • +
+
+
+ Push +
+
+

+ Any change to a service’s running software or its configuration. +

+
+
+
+
+

+ Why Monitor? +

+

+ There are many reasons to monitor a system, including: +

+
+ +
+

+ How big is my database and how fast is it growing? How quickly is my daily-active user count growing? +

+
+
+ Comparing over time or experiment groups +
+
+

+ Are queries faster with Acme Bucket of Bytes 2.72 versus Ajax DB 3.14? How much better is my memcache hit rate with an extra node? Is my site slower than it was last week? +

+
+
+ Alerting +
+
+

+ Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon. +

+
+
+ Building dashboards +
+
+

+ Dashboards should answer basic questions about your service, and normally include some form of the four golden signals (discussed in The Four Golden Signals). +

+
+
+ Conducting ad hoc retrospective analysis (i.e., debugging) +
+
+

+ Our latency just shot up; what else happened around the same time? +

+
+
+

+ System monitoring is also helpful in supplying raw input into business analytics and in facilitating analysis of security breaches. Because this book focuses on the engineering domains in which SRE has particular expertise, we won’t discuss these applications of monitoring here. +

+

+ Monitoring and alerting enables a system to tell us when it’s broken, or perhaps to tell us what’s about to break. When the system isn’t able to automatically fix itself, we want a human to investigate the alert, determine if there’s a real problem at hand, mitigate the problem, and determine the root cause of the problem. Unless you’re performing security auditing on very narrowly scoped components of a system, you should never trigger an alert simply because "something seems a bit weird." +

+

+ Paging a human is a quite expensive use of an employee’s time. If an employee is at work, a page interrupts their workflow. If the employee is at home, a page interrupts their personal time, and perhaps even their sleep. When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a "real" page that’s masked by the noise. Outages can be prolonged because other noise interferes with a rapid diagnosis and fix. Effective alerting systems have good signal and very low noise. +

+
+
+

+ Setting Reasonable Expectations for Monitoring +

+

+ Monitoring a complex application is a significant engineering endeavor in and of itself. Even with substantial existing infrastructure for instrumentation, collection, display, and alerting in place, a Google SRE team with 10–12 members typically has one or sometimes two members whose primary assignment is to build and maintain monitoring systems for their service. This number has decreased over time as we generalize and centralize common monitoring infrastructure, but every SRE team typically has at least one “monitoring person.” (That being said, while it can be fun to have access to traffic graph dashboards and the like, SRE teams carefully avoid any situation that requires someone to “stare at a screen to watch for problems.”) +

+

+ In general, Google has trended toward simpler and faster monitoring systems, with better tools for post hoc analysis. We avoid "magic" systems that try to learn thresholds or automatically detect causality. Rules that detect unexpected changes in end-user request rates are one counterexample; while these rules are still kept as simple as possible, they give a very quick detection of a very simple, specific, severe anomaly. Other uses of monitoring data such as capacity planning and traffic prediction can tolerate more fragility, and thus, more complexity. Observational experiments conducted over a very long time horizon (months or years) with a low sampling rate (hours or days) can also often tolerate more fragility because occasional missed samples won’t hide a long-running trend. +

+

+ Google SRE has experienced only limited success with complex dependency hierarchies. We seldom use rules such as, "If I know the database is slow, alert for a slow database; otherwise, alert for the website being generally slow." Dependency-reliant rules usually pertain to very stable parts of our system, such as our system for draining user traffic away from a datacenter. For example, "If a datacenter is drained, then don’t alert me on its latency" is one common datacenter alerting rule. Few teams at Google maintain complex dependency hierarchies because our infrastructure has a steady rate of continuous refactoring. +

+

+ Some of the ideas described in this chapter are still aspirational: there is always room to move more rapidly from symptom to root cause(s), especially in ever-changing systems. So while this chapter sets out some goals for monitoring systems, and some ways to achieve these goals, it’s important that monitoring systems—especially the critical path from the onset of a production problem, through a page to a human, through basic triage and deep debugging—be kept simple and comprehensible by everyone on the team. +

+

+ Similarly, to keep noise low and signal high, the elements of your monitoring system that direct to a pager need to be very simple and robust. Rules that generate alerts for humans should be simple to understand and represent a clear failure. +

+
+
+

+ Symptoms Versus Causes +

+

+ Your monitoring system should address two questions: what’s broken, and why? +

+

+ The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. Table 6-1 lists some hypothetical symptoms and corresponding causes. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Table 6-1. Example symptoms and causes +
+ Symptom + + Cause +
+

+ I’m serving HTTP 500s or 404s +

+
+

+ Database servers are refusing connections +

+
+

+ My responses are slow +

+
+

+ CPUs are overloaded by a bogosort, or an Ethernet cable is crimped under a rack, visible as partial packet loss +

+
+

+ Users in Antarctica aren’t receiving animated cat GIFs +

+
+

+ Your Content Distribution Network hates scientists and felines, and thus blacklisted some client IPs +

+
+

+ Private content is world-readable +

+
+

+ A new software push caused ACLs to be forgotten and allowed all requests +

+
+

+ "What" versus "why" is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise. +

+
+
+

+ Black-Box Versus White-Box +

+

+ We combine heavy use of white-box monitoring with modest but critical uses of black-box monitoring. The simplest way to think about black-box monitoring versus white-box monitoring is that black-box monitoring is symptom-oriented and represents active—not predicted—problems: "The system isn’t working correctly, right now." White-box monitoring depends on the ability to inspect the innards of the system, such as logs or HTTP endpoints, with instrumentation. White-box monitoring therefore allows detection of imminent problems, failures masked by retries, and so forth. +

+

+ Note that in a multilayered system, one person’s symptom is another person’s cause. For example, suppose that a database’s performance is slow. Slow database reads are a symptom for the database SRE who detects them. However, for the frontend SRE observing a slow website, the same slow database reads are a cause. Therefore, white-box monitoring is sometimes symptom-oriented, and sometimes cause-oriented, depending on just how informative your white-box is. +

+

+ When collecting telemetry for debugging, white-box monitoring is essential. If web servers seem slow on database-heavy requests, you need to know both how fast the web server perceives the database to be, and how fast the database believes itself to be. Otherwise, you can’t distinguish an actually slow database server from a network problem between your web server and your database. +

+

+ For paging, black-box monitoring has the key benefit of forcing discipline to only nag a human when a problem is both already ongoing and contributing to real symptoms. On the other hand, for not-yet-occurring but imminent problems, black-box monitoring is fairly useless. +

+
+
+

+ The Four Golden Signals +

+

+ The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four. +

+
+
+ Latency +
+
+

+ The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors. +

+
+
+ Traffic +
+
+

+ A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per second. +

+
+
+ Errors +
+
+

+ The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content. +

+
+
+ Saturation +
+
+

+ How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. +

+

+ In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. +

+

+ Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours." +

+
+
+

+ If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring. +

+
+
+

+ Worrying About Your Tail (or, Instrumentation and Performance) +

+

+ When building a monitoring system from scratch, it’s tempting to design a system based upon the mean of some quantity: the mean latency, the mean CPU usage of your nodes, or the mean fullness of your databases. The danger presented by the latter two cases is obvious: CPUs and databases can easily be utilized in a very imbalanced way. The same holds for latency. If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.23 If your users depend on several such web services to render their page, the 99th percentile of one backend can easily become the median response of your frontend. +

+

+ The simplest way to differentiate between a slow average and a very slow "tail" of requests is to collect request counts bucketed by latencies (suitable for rendering a histogram), rather than actual latencies: how many requests did I serve that took between 0 ms and 10 ms, between 10 ms and 30 ms, between 30 ms and 100 ms, between 100 ms and 300 ms, and so on? Distributing the histogram boundaries approximately exponentially (in this case by factors of roughly 3) is often an easy way to visualize the distribution of your requests. +

+
+
+

+ Choosing an Appropriate Resolution for Measurements +

+

+ Different aspects of a system should be measured with different levels of granularity. For example: +

+
    +
  • Observing CPU load over the time span of a minute won’t reveal even quite long-lived spikes that drive high tail latencies. +
  • +
  • On the other hand, for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime), probing for a 200 (success) status more than once or twice a minute is probably unnecessarily frequent. +
  • +
  • Similarly, checking hard drive fullness for a service targeting 99.9% availability more than once every 1–2 minutes is probably unnecessary. +
  • +
+

+ Take care in how you structure the granularity of your measurements. Collecting per-second measurements of CPU load might yield interesting data, but such frequent measurements may be very expensive to collect, store, and analyze. If your monitoring goal calls for high resolution but doesn’t require extremely low latency, you can reduce these costs by performing internal sampling on the server, then configuring an external system to collect and aggregate that distribution over time or across servers. You might: +

+
    +
  1. Record the current CPU utilization each second. +
  2. +
  3. Using buckets of 5% granularity, increment the appropriate CPU utilization bucket each second. +
  4. +
  5. Aggregate those values every minute. +
  6. +
+

+ This strategy allows you to observe brief CPU hotspots without incurring very high cost due to collection and retention. +

+
+
+

+ As Simple as Possible, No Simpler +

+

+ Piling all these requirements on top of each other can add up to a very complex monitoring system—your system might end up with the following levels of complexity: +

+
    +
  • Alerts on different latency thresholds, at different percentiles, on all kinds of different metrics +
  • +
  • Extra code to detect and expose possible causes +
  • +
  • Associated dashboards for each of these possible causes +
  • +
+

+ The sources of potential complexity are never-ending. Like all software systems, monitoring can become so complex that it’s fragile, complicated to change, and a maintenance burden. +

+

+ Therefore, design your monitoring system with an eye toward simplicity. In choosing what to monitor, keep the following guidelines in mind: +

+
    +
  • The rules that catch real incidents most often should be as simple, predictable, and reliable as possible. +
  • +
  • Data collection, aggregation, and alerting configuration that is rarely exercised (e.g., less than once a quarter for some SRE teams) should be up for removal. +
  • +
  • Signals that are collected, but not exposed in any prebaked dashboard nor used by any alert, are candidates for removal. +
  • +
+

+ In Google’s experience, basic collection and aggregation of metrics, paired with alerting and dashboards, has worked well as a relatively standalone system. (In fact Google’s monitoring system is broken up into several binaries, but typically people learn about all aspects of these binaries.) It can be tempting to combine monitoring with other aspects of inspecting complex systems, such as detailed system profiling, single-process debugging, tracking details about exceptions or crashes, load testing, log collection and analysis, or traffic inspection. While most of these subjects share commonalities with basic monitoring, blending together too many results in overly complex and fragile systems. As in many other aspects of software engineering, maintaining distinct systems with clear, simple, loosely coupled points of integration is a better strategy (for example, using web APIs for pulling summary data in a format that can remain constant over an extended period of time). +

+
+
+

+ Tying These Principles Together +

+

+ The principles discussed in this chapter can be tied together into a philosophy on monitoring and alerting that’s widely endorsed and followed within Google SRE teams. While this monitoring philosophy is a bit aspirational, it’s a good starting point for writing or reviewing a new alert, and it can help your organization ask the right questions, regardless of the size of your organization or the complexity of your service or system. +

+

+ When creating rules for monitoring and alerting, asking the following questions can help you avoid false positives and pager burnout:24 +

+
    +
  • Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?25 +
  • +
  • Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario? +
  • +
  • Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out? +
  • +
  • Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround? +
  • +
  • Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary? +
  • +
+

+ These questions reflect a fundamental philosophy on pages and pagers: +

+
    +
  • Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued. +
  • +
  • Every page should be actionable. +
  • +
  • Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page. +
  • +
  • Pages should be about a novel problem or an event that hasn’t been seen before. +
  • +
+

+ Such a perspective dissipates certain distinctions: if a page satisfies the preceding four bullets, it’s irrelevant whether the page is triggered by white-box or black-box monitoring. This perspective also amplifies certain distinctions: it’s better to spend much more effort on catching symptoms than causes; when it comes to causes, only worry about very definite, very imminent causes. +

+
+
+

+ Monitoring for the Long Term +

+

+ In modern production systems, monitoring systems track an ever-evolving system with changing software architecture, load characteristics, and performance targets. An alert that’s currently exceptionally rare and hard to automate might become frequent, perhaps even meriting a hacked-together script to resolve it. At this point, someone should find and eliminate the root causes of the problem; if such resolution isn’t possible, the alert response deserves to be fully automated. +

+

+ It’s important that decisions about monitoring be made with long-term goals in mind. Every page that happens today distracts a human from improving the system for tomorrow, so there is often a case for taking a short-term hit to availability or performance in order to improve the long-term outlook for the system. Let’s take a look at two case studies that illustrate this trade-off. +

+
+

+ Bigtable SRE: A Tale of Over-Alerting +

+

+ Google’s internal infrastructure is typically offered and measured against a service level objective (SLO; see Service Level Objectives). Many years ago, the Bigtable service’s SLO was based on a synthetic well-behaved client’s mean performance. Because of problems in Bigtable and lower layers of the storage stack, the mean performance was driven by a "large" tail: the worst 5% of requests were often significantly slower than the rest. +

+

+ Email alerts were triggered as the SLO approached, and paging alerts were triggered when the SLO was exceeded. Both types of alerts were firing voluminously, consuming unacceptable amounts of engineering time: the team spent significant amounts of time triaging the alerts to find the few that were really actionable, and we often missed the problems that actually affected users, because so few of them did. Many of the pages were non-urgent, due to well-understood problems in the infrastructure, and had either rote responses or received no response. +

+

+ To remedy the situation, the team used a three-pronged approach: while making great efforts to improve the performance of Bigtable, we also temporarily dialed back our SLO target, using the 75th percentile request latency. We also disabled email alerts, as there were so many that spending time diagnosing them was infeasible. +

+

+ This strategy gave us enough breathing room to actually fix the longer-term problems in Bigtable and the lower layers of the storage stack, rather than constantly fixing tactical problems. On-call engineers could actually accomplish work when they weren’t being kept up by pages at all hours. Ultimately, temporarily backing off on our alerts allowed us to make faster progress toward a better service. +

+
+
+

+ Gmail: Predictable, Scriptable Responses from Humans +

+

+ In the very early days of Gmail, the service was built on a retrofitted distributed process management system called Workqueue, which was originally created for batch processing of pieces of the search index. Workqueue was "adapted" to long-lived processes and subsequently applied to Gmail, but certain bugs in the relatively opaque codebase in the scheduler proved hard to beat. +

+

+ At that time, the Gmail monitoring was structured such that alerts fired when individual tasks were “de-scheduled” by Workqueue. This setup was less than ideal because even at that time, Gmail had many, many thousands of tasks, each task representing a fraction of a percent of our users. We cared deeply about providing a good user experience for Gmail users, but such an alerting setup was unmaintainable. +

+

+ To address this problem, Gmail SRE built a tool that helped “poke” the scheduler in just the right way to minimize impact to users. The team had several discussions about whether or not we should simply automate the entire loop from detecting the problem to nudging the rescheduler, until a better long-term solution was achieved, but some worried this kind of workaround would delay a real fix. +

+

+ This kind of tension is common within a team, and often reflects an underlying mistrust of the team’s self-discipline: while some team members want to implement a “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it’s easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes. Managers and technical leaders play a key role in implementing true, long-term fixes by supporting and prioritizing potentially time-consuming long-term fixes even when the initial “pain” of paging subsides. +

+

+ Pages with rote, algorithmic responses should be a red flag. Unwillingness on the part of your team to automate such pages implies that the team lacks confidence that they can clean up their technical debt. This is a major problem worth escalating. +

+
+
+

+ The Long Run +

+

+ A common theme connects the previous examples of Bigtable and Gmail: a tension between short-term and long-term availability. Often, sheer force of effort can help a rickety system achieve high availability, but this path is usually short-lived and fraught with burnout and dependence on a small number of heroic team members. Taking a controlled, short-term decrease in availability is often a painful, but strategic trade for the long-run stability of the system. It’s important not to think of every page as an event in isolation, but to consider whether the overall level of paging leads toward a healthy, appropriately available system with a healthy, viable team and long-term outlook. We review statistics about page frequency (usually expressed as incidents per shift, where an incident might be composed of a few related pages) in quarterly reports with management, ensuring that decision makers are kept up to date on the pager load and overall health of their teams. +

+
+
+
+

+ Conclusion +

+

+ A healthy monitoring and alerting pipeline is simple and easy to reason about. It focuses primarily on symptoms for paging, reserving cause-oriented heuristics to serve as aids to debugging problems. Monitoring symptoms is easier the further "up" your stack you monitor, though monitoring saturation and performance of subsystems such as databases often must be performed directly on the subsystem itself. Email alerts are of very limited value and tend to easily become overrun with noise; instead, you should favor a dashboard that monitors all ongoing subcritical problems for the sort of information that typically ends up in email alerts. A dashboard might also be paired with a log, in order to analyze historical correlations. +

+

+ Over the long haul, achieving a successful on-call rotation and product includes choosing to alert on symptoms or imminent real problems, adapting your targets to goals that are actually achievable, and making sure that your monitoring supports rapid diagnosis. +

+
+
+

+ 22Sometimes known as "alert spam," as they are rarely read or acted on. +

+

+ 23If 1% of your requests are 50x the average, it means that the rest of your requests are about twice as fast as the average. But if you’re not measuring your distribution, the idea that most of your requests are near the mean is just hopeful thinking. +

+

+ 24See Applying Cardiac Alarm Management Techniques to Your On-Call [Hol14] for an example of alert fatigue in another context. +

+

+ 25Zero-redundancy (N + 0) situations count as imminent, as do "nearly full" parts of your service! For more details about the concept of redundancy, see https://en.wikipedia.org/wiki/N%2B1_redundancy. +

+
+
+
+
+ +
+ + + + + + diff --git a/src/full_text_parser/readability/tests.rs b/src/full_text_parser/readability/tests.rs index eaf16fb..7df63c6 100644 --- a/src/full_text_parser/readability/tests.rs +++ b/src/full_text_parser/readability/tests.rs @@ -201,6 +201,21 @@ async fn firefox_nightly_blog() { run_test("firefox-nightly-blog").await } +#[tokio::test] +async fn folha() { + run_test("folha").await +} + +#[tokio::test] +async fn gmw() { + run_test("gmw").await +} + +#[tokio::test] +async fn google_sre_book_1() { + run_test("google-sre-book-1").await +} + #[tokio::test] async fn webmd_1() { run_test("webmd-1").await