[Summer Code Camp] 题目申请

[Ken]

1. [curvefs/mds] Optimizing unit tests (Easy)

[curvefs/mds] Optimizing unit tests

2. curvefs/client: cancel the warm-up task in the warm-up queue (Medium)

curvefs/client: cancel the warm-up task in the warm-up queue

3. curvefs/client:warmup list and stop (Medium)

issues/2244

如果可以的话，希望可以选到一题medium配上unit tests optimization，谢谢！

[Ken]

[题目思路] 3. curvefs/client:warmup list and stop (Medium level)

curvefs/client:warmup list and stop

1. Ensure every path in the file list is valid

• [NEW] add.go/verifyFilelist - create a new function verifyFilelist, which reuses most of the convertFilelist function (except for the path replacement part)
• [DEL] add.go/convertFilelist - delete it since no one else will use it anymore.
• [MODIFY] add.go/RunCommand - use the newly created function: verifyFilelist when we know it is a warm-up add filelist operation

2. Pass the prefix of the user path and curvefs path to the client

• [MOD] add.go/RunCommand - append two more parameters to value: mountpoint.Root, mountpoint.MountPoint

3. Extract the prefix of the user path and curvefs path from the passed-in value

• [MOD] curve_fuse_op.cpp/Warmup - extract mountpoint.Root and mountpoint.MountPoint from values[4], values[5]
• [MOD] common.h/kWarmupOpNum - change the number from 4 to 6

4. Forward mountpoint.Root and mountpoint.MountPoint all the way to where they will be used

• [MOD] curve_fuse_op.cpp/Warmup - include mountpoint.Root and mountpoint.MountPoint as parameters when calling AddWarmupTask

• [MOD] curve_fuse_op.cpp/AddWarmupTask - change its signature to receive mountpoint.Root and mountpoint.MountPoint

• [MOD] curve_fuse_op.cpp/AddWarmupTask - include mountpoint.Root and mountpoint.MountPoint as parameters when calling g_ClientInstance->PutWarmFilelistTask

• [MOD] fuse_client.h/PutWarmFilelistTask - change its signature to receive mountpoint.Root and mountpoint.MountPoint

• [MOD] fuse_client.h/PutWarmFilelistTask - include mountpoint.Root and mountpoint.MountPoint as parameters when calling warmupManager_->AddWarmupFilelist

• [MOD] warmup_manager.cpp/AddWarmupFilelist - change its signature to receive mountpoint.Root and mountpoint.MountPoint

• [MOD] warmup_manager.cpp/AddWarmupFilelist - include mountpoint.Root and mountpoint.MountPoint as parameters when calling warmupFilelistDeque_.emplace_back

• [MOD] warmup_manager.h/WarmupFile - create two new members: mountpoint.Root and mountpoint.MountPoint for WarmupFile class, and update WarmupFile’s constructor

5. Replace the user path with the curvefs path

• [MOD] warmup_manager.cpp/GetWarmupList - extract mountpoint.Root and mountpoint.MountPoint from filelist, then update each line of file by replacing the mountpoint.MountPoint with mountpoint.Root

6. Verify the change is valid

(I couldn’t find any existing test cases related to functions I will change, so I suppose there are no relevant test cases available at the moment?)

• Manually run the tool to see if the update works

[caoxianfei]

ok, @Cyber-SiKu has already replied to you on github, then you can take a look. issue/2244

[Ken]

[题目思路] 2. curvefs/client: cancel the warm-up task in the warm-up queue (Medium level)

curvefs/client: cancel the warm-up task in the warm-up queue

1. Implement cancel command on CLI

• [NEW] cli/command/curefs/warmup/cancel.go

• [MOD] cli/command/curefs/warmup/warmup.go

2. Add a new operation type - cancellation

• [MOD] curve_fuse_op.cpp/Warmup

• [MOD] common.cpp/GetWarmupOpType

• [MOD] common.h/WarmupOpType

3. Create wrappers for CancelWarmupTask

• [NEW] curve_fuse_op.cpp/CancelWarmupTask

• [NEW] fuse_client.h/CancelWarmFileTask

• [NEW] fuse_client.h/CancelWarmFilelistTask

4. Implement logics for CancelWarmFileTask & CancelWarmFilelistTask

• [NEW] warmup_manager.h/CancelWarmFileTask

• [NEW] warmup_manager.h/CancelWarmFilelistTask

• [NEW] warmup_manager.cpp/CancelWarmFileTask

  • inode2FetchDentryPool_.get(key).stop()
  • inode2FetchDentryPool_.erase(inode2FetchDentryPool_.get(key))
  • warmupInodesDeque_.remove(key)
  • inode2FetchS3ObjectsPool_.get(key).Stop()
  • inode2FetchS3ObjectsPool_.erase(inode2FetchS3ObjectsPool_.get(key))
  • inode2Progress_.remove(key)

• [NEW] warmup_manager.cpp/CancelWarmFilelistTask

  • warmupFilelistDeque_.remove(key)

5. Verify the implementation

Currently, I don’t have a concrete plan for the testing because I am still trying to understand many key concepts related to it; such as how to apply the mock function to the thread pool blocking: curvefs/client: cancel the warm-up task in the warm-up queue · Issue #2019 · opencurve/curve · GitHub.

Therefore, I won’t provide a detailed walkthrough right now, but I will update it later once I figure everything out.

• [NEW] test_fuse_s3_client.cpp/TEST_F(TestFuseS3Client, warmUp_Cancel_filel)

  • Add a warm-up task
  • Somehow block it
  • Verify that warmup_manger has corresponding entries in inode2FetchDentryPool_, warmupInodesDeque_, inode2FetchS3ObjectsPool_, inode2Progress_
  • Perform the cancellation
  • Verify that warmup_manger does not have corresponding entries in inode2FetchDentryPool_, warmupInodesDeque_, inode2FetchS3ObjectsPool_, inode2Progress_

• [NEW] test_fuse_s3_client.cpp/TEST_F(TestFuseS3Client, warmUp_Cancel_filelist)

  • Have no idea for now

[Ken]

[题目思路] 1. [curvefs/mds] Optimizing unit tests (Easy level)

[curvefs/mds] Optimizing unit tests

I am planning to refactor all the test cases in these two files, and I have outlined the procedure in the following steps:

• Split the gigantic test case into smaller and single-purpose cases.
• Group related unit test cases into a test suite.
• Rename the unit test cases using a more meaningful, camel-style name like CleanUpOnCreateFsWithFailedDentryCreation.
• Create sub-class fixtures in each test suite to group common ground for multiple test cases, such as FSManagerTestForCleanUp inheriting from FSManagerTest and overriding it.
• Remove assertions that are less relevant to the test case’s main objective.
  For instance, I will consider removing EXPECT_CALL for mockCliService2_, as it might make the test case more complex and distract from its essential objectives.

---

Below is a prototype of the CleanUpWhenCreateFsFail test case and my initial approach to working on it.

+TEST_SUITE(CleanUpWhenCreateFsFail);

+class FSManagerTestForCleanUp : public ::FSManagerTest {
+ protected:
+    void SetUp() override {
+        FSManagerTest::SetUp();

+        std::string addr = addr_;
+        std::string leader = addr_ + ":0";
+        std::set<std::string> addrs{addr};
+        std::string fsName = "fs";
+        uint64_t blockSize = 4096;
+        bool enableSumInDir = false;
+        Volume volume;
+
+        volume.set_blocksize(blockSize);
+        volume.set_volumename("volume");
+        volume.set_user("user");
+        volume.add_cluster(addr_);
+
+        FsInfo fsInfo;
+        FsDetail fsDetail;
+        fsDetail.set_allocated_volume(new Volume(volume));
+
+        req.set_fsname(fsName);
+        req.set_blocksize(blockSize);
+        req.set_fstype(FSType::TYPE_VOLUME);
+        req.set_allocated_fsdetail(new FsDetail(fsDetail));
+        req.set_enablesumindir(enableSumInDir);
+        req.set_owner("test");
+        req.set_capacity((uint64_t)100 * 1024 * 1024 * 1024);
+        req.set_recycletimehour(10);
+
+        getLeaderResponse.mutable_leader()->set_address(leader);
+        createRootInodeResponse.set_statuscode(MetaStatusCode::OK);
+        deleteInodeResponse.set_statuscode(MetaStatusCode::OK);
+
-        EXPECT_CALL(*topoManager_, CreatePartitionsAndGetMinPartition(_, _))
-            .WillOnce(Return(TopoStatusCode::TOPO_OK));
-        EXPECT_CALL(*topoManager_, GetCopysetMembers(_, _, _))
-            .WillOnce(DoAll(SetArgPointee<2>(addrs),
-                            Return(TopoStatusCode::TOPO_OK)));
-        EXPECT_CALL(mockMetaserverService_, CreateRootInode(_, _, _, _))
-            .WillOnce(DoAll(SetArgPointee<2>(createRootInodeResponse),
-                            Invoke(RpcService<CreateRootInodeRequest,
-                                              CreateRootInodeResponse>)));
-        EXPECT_CALL(mockMetaserverService_, CreateManageInode(_, _, _, _))
-            .WillOnce(DoAll(SetArgPointee<2>(createManageInodeResponse),
-                            Invoke(RpcService<CreateManageInodeRequest,
-                                              CreateManageInodeResponse>)));
+
-        EXPECT_CALL(*topoManager_, DeletePartition(_))
-            .WillOnce(Return(TopoStatusCode::TOPO_OK));
+
-        EXPECT_CALL(mockCliService2_, GetLeader(_, _, _, _))
-            .Times(AnyNumber())
-            .WillRepeatedly(DoAll(
-                SetArgPointee<2>(getLeaderResponse),
-                Invoke(RpcService<GetLeaderRequest2, GetLeaderResponse2>)));
+    }

+ protected:
+    std::string addr_;
+    std::string leader;
+    std::set<std::string> addrs;
+    std::shared_ptr<GetLeaderResponse2> getLeaderResponse;
+    std::shared_ptr<CreateRootInodeResponse> createRootInodeResponse;
+    std::shared_ptr<DeleteInodeResponse> deleteInodeResponse;
+    std::shared_ptr<CreateManageInodeResponse> createManageInodeResponse;
+    std::shared_ptr<FsInfoWrapper> wrapper;
+
+}

+TEST_F(FSManagerTestForCleanUp, CleanUpOnCreateFsWithFailedInodeCreation) {
     createManageInodeResponse.set_statuscode(MetaStatusCode::UNKNOWN_ERROR);
+
     EXPECT_CALL(mockMetaserverService_, DeleteInode(_, _, _, _))
         .WillOnce(
             DoAll(SetArgPointee<2>(deleteInodeResponse),
                   Invoke(RpcService<DeleteInodeRequest, DeleteInodeResponse>)));
     ASSERT_EQ(FSStatusCode::INSERT_MANAGE_INODE_FAIL,
               fsManager_->CreateFs(&req, &fsInfo));
     ASSERT_EQ(FSStatusCode::NOT_FOUND, fsStorage_->Get(fsName, &wrapper));
+}

+TEST_F(FSManagerTestForCleanUp, CleanUpOnCreateFsWithFailedDentryCreation) {
     createManageInodeResponse.set_statuscode(MetaStatusCode::OK);

     CreateDentryResponse createDentryResponse;
     createDentryResponse.set_statuscode(MetaStatusCode::UNKNOWN_ERROR);
         .WillOnce(DoAll(
             SetArgPointee<2>(createDentryResponse),
             Invoke(RpcService<CreateDentryRequest, CreateDentryResponse>)));
+
     EXPECT_CALL(mockMetaserverService_, DeleteInode(_, _, _, _))
         .Times(2)
         .WillRepeatedly(
             DoAll(SetArgPointee<2>(deleteInodeResponse),
                   Invoke(RpcService<DeleteInodeRequest, DeleteInodeResponse>)));

     ASSERT_EQ(FSStatusCode::INSERT_DENTRY_FAIL,
               fsManager_->CreateFs(&req, &fsInfo));
     ASSERT_EQ(FSStatusCode::NOT_FOUND, fsStorage_->Get(fsName, &wrapper));
 }

+TEST_SUITE_END();